SelfGNN: Self-supervised Graph Neural Networks without explicit negative sampling
Abstract
Real world data is mostly unlabeled or only few instances are labeled. Manually labeling data is a very expensive and daunting task. This calls for unsupervised learning techniques that are powerful enough to achieve comparable results as semi-supervised/supervised techniques. Contrastive self-supervised learning has emerged as a powerful direction, in some cases outperforming supervised techniques. In this study, we propose, SelfGNN, a novel contrastive self-supervised graph neural network (GNN) without relying on explicit contrastive terms. We leverage Batch Normalization, which introduces implicit contrastive terms, without sacrificing performance. Furthermore, as data augmentation is key in contrastive learning, we introduce four feature augmentation (FA) techniques for graphs. Though graph topological augmentation (TA) is commonly used, our empirical findings show that FA perform as good as TA. Moreover, FA incurs no computational overhead, unlike TA, which often has $O(N^3)$ time complexity, $N$ -- number of nodes. Our empirical evaluation on seven publicly available real-world data shows that, SelfGNN is powerful and leads to a performance comparable with SOTA supervised GNNs and always better than SOTA semi-supervised and unsupervised GNNs. The source code is available at https://github.com/zekarias-tilahun/SelfGNN.
Benchmark Results
Tested on 4 datasets 5 baseline values reported too low
Tables show presented results in global context (best 3 papers + GCN + MLP).
Classic
| # | Model | Accuracy ↑ | Paper |
|---|---|---|---|
| 1 | GraphSAGE+CNA | 0.9418
±0.0033 | |
| 2 | GCN+CNA | 0.9366
±0.0048 | |
| 3 | TransformerConv | 0.9350
±0.0058 | |
| 21 | GCN | 0.9004 | |
| 177 | DGI | 0.8630
±0.0020 best reported | best at pub: Deep Learning on Graphs: A Survey (2018) best today: Oversmoothing: A Nightmare for Graph Contrastive Learning? (2023) |
| 610 | SelfBFA | 0.8100
±0.0018 | This paper |
| 694 | SelfHK | 0.7901
±0.0031 | This paper |
| 696 | SelfPPR | 0.7898
±0.0025 | This paper |
| 722 | SelfKatz | 0.7771
±0.0034 | This paper |
| 724 | MLP | 0.7768 | |
| 770 | MvGrlPPR | 0.7530
±0.0072 | This paper |
| 780 | MvGrlHK | 0.7450
±0.0071 | This paper |
| 781 | MvGrlKatz | 0.7450
±0.0060 | This paper |
| # | Model | Accuracy ↑ | Paper |
|---|---|---|---|
| 1 | CNA | 0.9575
±0.0058 | |
| 2 | IS-GIB | 0.9390
±0.0187 | |
| 3 | EERM | 0.9112
±0.0145 | |
| 44 | GCN | 0.8033
±0.0088 | |
| 66 | DGI | 0.7890
±0.0020 best reported | best at pub: Deep Learning on Graphs: A Survey (2018) best today: Oversmoothing: A Nightmare for Graph Contrastive Learning? (2023) |
| 350 | MLP | 0.7289
±0.0010 | |
| 685 | SelfPPR | 0.6808
±0.0048 | This paper |
| 686 | SelfHK | 0.6801
±0.0043 | This paper |
| 717 | SelfBFA | 0.6719
±0.0037 | This paper |
| 751 | SelfKatz | 0.6614
±0.0032 | This paper |
| 820 | MvGrlKatz | 0.6150 | This paper |
| 843 | MvGrlPPR | 0.5820 | This paper |
| 845 | MvGrlHK | 0.5750
±0.0007 | This paper |
| # | Model | Accuracy ↑ | Paper |
|---|---|---|---|
| 1 | LGTL | 0.9518
±0.0011 | |
| 2 | MixHop (LO) | 0.9513
±0.0038 | |
| 3 | GraphSAGE (LO) | 0.9511
±0.0036 | |
| 18 | GCN | 0.9296
±0.0015 best reported | best at pub: Tackling Over-Smoothing for General Graph Convolutional Networks (2020) |
| 20 | GAT | 0.9233
±0.0018 best reported | best at pub: Tackling Over-Smoothing for General Graph Convolutional Networks (2020) |
| 116 | SAGE | 0.8881
±0.0015 | |
| 125 | GraphSAGE | 0.8864
±0.0030 best reported | best at pub: Tackling Over-Smoothing for General Graph Convolutional Networks (2020) best today: Non-backtracking Graph Neural Networks (TMLR 2023) |
| 210 | MLP | 0.8690
±0.0074 | |
| 311 | SelfGNN PPR (GAT) | 0.8270
±0.0001 | This paper |
| 314 | SelfGNN hk (GAT) | 0.8260
±0.0001 | This paper |
| 322 | SelfGNN Standard (GCN encoder) | 0.8230
±0.0001 | This paper |
| 358 | SelfGNN Split (GCN encoder) | 0.8130
±0.0001 | This paper |
| 374 | SelfGNN Katz (GCN encoder) | 0.8100
±0.0001 | This paper |
| 737 | SelfGNN LDP (GCN encoder) | 0.7710
±0.0001 | This paper |
| 770 | SelfGNN Paste (GAT) | 0.7610
±0.0001 | This paper |
| # | Model | Accuracy ↑ | Paper |
|---|---|---|---|
| 1 | GrokFormer | 0.9831
±0.0018 | |
| 2 | PolyFormer | 0.9808
±0.0027 | |
| 3 | SGFormer | 0.9796
±0.0081 | |
| 6 | GAT | 0.9620
±0.0010 | prev best: Graph Contrastive Learning with Adaptive Augmentation (WWW 2020) best today: GrokFormer: Graph Fourier Kolmogorov-Arnold Transformers (ICML 2024) |
| 9 | GCN | 0.9650
±0.0010 | prev best: Graph Contrastive Learning with Adaptive Augmentation (WWW 2020) best today: GrokFormer: Graph Fourier Kolmogorov-Arnold Transformers (ICML 2024) |
| 17 | SAGE | 0.9739
±0.0018 | |
| 58 | MLP | 0.9597
±0.0007 | |
| 121 | SelfGNN Split | 0.9550
±0.0030 | This paper |
| 122 | SelfGNN Paste | 0.9550
±0.0020 | This paper |
| 160 | SelfGNN LDP | 0.9500
±0.0020 | This paper |
| 169 | SelfGNN Standard | 0.9480 | This paper |
