WWW 2021

CogDL: A Comprehensive Library for Graph Deep Learning

Yukuo Cen, Zhenyu Hou, Yan Wang, Qibin Chen, Yizhen Luo, Zhongming Yu, Hengrui Zhang, Xingcheng Yao, Aohan Zeng, Shiguang Guo, Yuxiao Dong, Yang Yang, Peng Zhang, Guohao Dai, Yu Wang, Chang Zhou, Hongxia Yang, Jie Tang*

Main figure for CogDL: A Comprehensive Library for Graph Deep Learning

Abstract

Graph neural networks (GNNs) have attracted tremendous attention from the graph learning community in recent years. It has been widely adopted in various real-world applications from diverse domains, such as social networks and biological graphs. The research and applications of graph deep learning present new challenges, including the sparse nature of graph data, complicated training of GNNs, and non-standard evaluation of graph tasks. To tackle the issues, we present CogDL, a comprehensive library for graph deep learning that allows researchers and practitioners to conduct experiments, compare methods, and build applications with ease and efficiency. In CogDL, we propose a unified design for the training and evaluation of GNN models for various graph tasks, making it unique among existing graph learning libraries. By utilizing this unified trainer, CogDL can optimize the GNN training loop with several training techniques, such as mixed precision training. Moreover, we develop efficient sparse operators for CogDL, enabling it to become the most competitive graph library for efficiency. Another important CogDL feature is its focus on ease of use with the aim of facilitating open and reproducible research of graph learning. We leverage CogDL to report and maintain benchmark results on fundamental graph tasks, which can be reproduced and directly used by the community.

Benchmark Results

Tested on 4 datasets 5 baseline values reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

Classic

#ModelF1 ↑Paper
1
GAMLP(JK)
0.9982 ±0.0001
2
GraphCON-GCN
0.9960
3
GCNII
actual #3 vs paper #70 67 places worse
0.9650 ±0.0020
actual #3 vs paper #70 67 places worse
paper 0.9650 ±0.0020
paper 0.9650 ±0.0020 Δ at pub 0.4580 Δ today 0.0303
4
GraphSAINT
actual #4 vs paper #11 7 places worse
0.9930 ±0.0010
actual #4 vs paper #11 7 places worse
paper 0.9930 ±0.0010
paper 0.9930 ±0.0010 Δ at pub 0.0120 Δ today 0.0020
21
GCN
actual #21 vs paper #108 87 places worse
0.9850
best reported
actual #21 vs paper #108 87 places worse
paper 0.7570 ±0.0010
best today: Graph-Coupled Oscillator Networks (ICML 2022)
paper 0.7570 ±0.0010 Δ at pub 0.0510 Δ today 0.2280
38
Cluster-SAGE
0.9790 ±0.0010
This paper
57
GAT
actual #57 vs paper #57
0.9732 ±0.0032
best reported
actual #57 vs paper #57
paper 0.9730 ±0.0020

Difference within reported uncertainty; not counted as underperformance.

paper 0.9730 ±0.0020 Δ today 0.0002
90
SGC
actual #90 vs paper #170 80 places worse
0.8920 ±0.0150
best reported
actual #90 vs paper #170 80 places worse
paper 0.4720 ±0.0010
paper 0.4720 ±0.0010 Δ today 0.4200
92
GIN
0.8724 ±0.0002
151
APPNP
0.6230 ±0.0010
This paper
168
PPRGo
0.4860 ±0.0010
This paper
173
MLP
0.4620

TU Dortmund

#ModelAccuracy ↑Paper
1
MSH-GNN
0.9910 ±0.0030
2
ID-GNN+BEC
0.9830 ±0.0360
3
SupCosine
0.9830 ±0.0250
39
GIN
actual #39 vs paper #72 33 places worse
0.9206
actual #39 vs paper #72 33 places worse
paper 0.9206
prev best: Provably Powerful Graph Networks (NeurIPS 2019)

Improvement within reported uncertainty; not highlighted.

paper 0.9206 Δ at pub 0.0266
63
GCN
0.9220 ±0.0440
190
SortPool
actual #190 vs paper #374 184 places worse
0.8725
actual #190 vs paper #374 184 places worse
paper 0.8725

Improvement within reported uncertainty; not highlighted.

paper 0.8725 Δ at pub 0.0285
239
DiffPool
actual #239 vs paper #446 207 places worse
0.8518
actual #239 vs paper #446 207 places worse
paper 0.8518

Improvement within reported uncertainty; not highlighted.

paper 0.8518 Δ at pub 0.0018 Δ today 0.0422
257
InfoGraph
actual #257 vs paper #279 22 places worse
0.8930 ±0.0514
best reported
actual #257 vs paper #279 22 places worse
paper 0.8895

Difference within reported uncertainty; not counted as underperformance.

paper 0.8895 Δ at pub 0.0005 Δ today 0.0035
365
DGK
actual #365 vs paper #436 71 places worse
0.8744 ±0.0272
best reported
actual #365 vs paper #436 71 places worse
paper 0.8558
best at pub: Provably Powerful Graph Networks (NeurIPS 2019)

Difference within reported uncertainty; not counted as underperformance.

paper 0.8558 Δ at pub 0.0186 Δ today 0.0186
403
MLP
0.8660 ±0.0495
407
graph2vec
actual #407 vs paper #504 97 places worse
0.8368
actual #407 vs paper #504 97 places worse
paper 0.8368

Improvement within reported uncertainty; not highlighted.

paper 0.8368 Δ at pub 0.0048 Δ today 0.0276
417
PATCHY_SAN
0.8612
This paper
425
DGCNN
actual #425 vs paper #512 87 places worse
0.8583 ±0.0116
best reported
actual #425 vs paper #512 87 places worse
paper 0.8333
623
SAGPool
actual #623 vs paper #753 130 places worse
0.7986 ±0.0236
best reported
actual #623 vs paper #753 130 places worse
paper 0.7173
paper 0.7173 Δ at pub 0.0677 Δ today 0.0813
#ModelAccuracy ↑Paper
1
WL-MLP
0.9954
2
GIN
actual #2 vs paper #147 145 places worse
0.9934
best reported
actual #2 vs paper #147 145 places worse
paper 0.8166
best at pub: Provably Powerful Graph Networks (NeurIPS 2019)
paper 0.8166 Δ at pub 0.0104 Δ today 0.1768
3
GCN
0.9727
5
DiffPool
actual #5 vs paper #495 490 places worse
0.9210 ±0.0260
best reported
actual #5 vs paper #495 490 places worse
paper 0.6909
paper 0.6909 Δ at pub 0.0864 Δ today 0.2301
131
GraphSAGE
0.8200 ±0.0100
259
SAGPool
actual #259 vs paper #452 193 places worse
0.7910 ±0.0300
best reported
actual #259 vs paper #452 193 places worse
paper 0.7287
paper 0.7287 Δ at pub 0.0501 Δ today 0.0623
338
InfoGraph
actual #338 vs paper #371 33 places worse
0.7664
actual #338 vs paper #371 33 places worse
paper 0.7664
paper 0.7664 Δ at pub 0.0571
373
DGCNN
actual #373 vs paper #519 146 places worse
0.7650 ±0.0190
best reported
actual #373 vs paper #519 146 places worse
paper 0.6596
paper 0.6596 Δ at pub 0.1054
406
graph2vec
actual #406 vs paper #461 55 places worse
0.7540 ±0.0120
best reported
actual #406 vs paper #461 55 places worse
paper 0.7185
paper 0.7185 Δ today 0.0355
415
SortPool
actual #415 vs paper #439 24 places worse
0.7399
actual #415 vs paper #439 24 places worse
paper 0.7399

Improvement within reported uncertainty; not highlighted.

paper 0.7399 Δ at pub 0.0174
484
PATCHY_SAN
0.6982
This paper
#ModelAccuracy ↑Paper
1
WL-MLP
0.9792
2
MSH-GNN
0.9640 ±0.0070
3
CoCN
0.8722 ±0.0013
6
graph2vec
0.8558
This paper
23
GCN
0.8260 ±0.0220
30
GIN
actual #30 vs paper #144 114 places worse
0.8220 ±0.0210
best reported
actual #30 vs paper #144 114 places worse
paper 0.7952

Difference within reported uncertainty; not counted as underperformance.

paper 0.7952 Δ at pub 0.0068 Δ today 0.0268
33
DiffPool
actual #33 vs paper #148 115 places worse
0.7927
actual #33 vs paper #148 115 places worse
paper 0.7927

Improvement within reported uncertainty; not highlighted.

paper 0.7927 Δ at pub 0.0037 Δ today 0.0286
125
SortPool
actual #125 vs paper #125
0.8007
actual #125 vs paper #125
paper 0.8007
paper 0.8007 Δ at pub 0.0237
146
InfoGraph
0.7940
This paper
151
MLP
0.7920 ±0.0240
191
DGCNN
actual #191 vs paper #194 3 places worse
0.7745
actual #191 vs paper #194 3 places worse
paper 0.7745
prev best: Graph U-Nets (2019)
paper 0.7745 Δ at pub 0.0369
258
PATCHY_SAN
0.7434
This paper
SOTA performance (not significantly different from the best) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green) Best at pub — reported value improved over the best known when published Improved at pub — later paper surpassed this result Best today — OK at publication; today's leaderboard best is elsewhere