ICLR 2021

AutoGL: A Library for Automated Graph Learning

Chaoyu Guan, Ziwei Zhang, Haoyang Li, Heng Chang, Zeyang Zhang, Yi Qin, Jiyan Jiang, Xin Wang, Wenwu Zhu

Main figure for AutoGL: A Library for Automated Graph Learning

Abstract

Recent years have witnessed an upsurge in research interests and applications of machine learning on graphs. However, manually designing the optimal machine learning algorithms for different graph datasets and tasks is inflexible, labor-intensive, and requires expert knowledge, limiting its adaptivity and applicability. Automated machine learning (AutoML) on graphs, aiming to automatically design the optimal machine learning algorithm for a given graph dataset and task, has received considerable attention. However, none of the existing libraries can fully support AutoML on graphs. To fill this gap, we present Automated Graph Learning (AutoGL), the first dedicated library for automated machine learning on graphs. AutoGL is open-source, easy to use, and flexible to be extended. Specifically, we propose a three-layer architecture, consisting of backends to interface with devices, a complete automated graph learning pipeline, and supported graph applications. The automated machine learning pipeline further contains five functional modules: auto feature engineering, neural architecture search, hyper-parameter optimization, model training, and auto ensemble, covering the majority of existing AutoML methods on graphs. For each module, we provide numerous state-of-the-art methods and flexible base classes and APIs, which allow easy usage and customization. We further provide experimental results to showcase the usage of our AutoGL library. We also present AutoGL-light, a lightweight version of AutoGL to facilitate customizing pipelines and enriching applications, as well as benchmarks for graph neural architecture search. The codes of AutoGL are publicly available at.

Proposed method: AutoGL

Benchmark Results

Tested on 5 datasets 9 baseline values reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

Classic

#ModelAccuracy ↑Paper
1
GraphSAGE+CNA
0.9418 ±0.0033
2
GCN+CNA
0.9366 ±0.0048
3
TransformerConv
0.9350 ±0.0058
21
GCN
actual #21 vs paper #599 578 places worse
0.9004
best reported
actual #21 vs paper #599 578 places worse
paper 0.8110 ±0.0090
paper 0.8110 ±0.0090 Δ at pub 0.0607 Δ today 0.0894
39
SAGE
0.8911 ±0.0012
44
GAT
actual #44 vs paper #486 442 places worse
0.8897 ±0.0014
best reported
actual #44 vs paper #486 442 places worse
paper 0.8230 ±0.0070
paper 0.8230 ±0.0070 Δ at pub 0.0660 Δ today 0.0667
141
GraphSAGE
actual #141 vs paper #779 638 places worse
0.8708 ±0.0085
best reported
actual #141 vs paper #779 638 places worse
paper 0.7450 ±0.0180
403
AutoGL
0.8320 ±0.0060
This paper
410
random (50 trials, GAT)
0.8320 ±0.0070
This paper
411
TPE (50 trials, GAT)
0.8320 ±0.0080
This paper
493
None (GAT)
0.8230 ±0.0070
This paper
511
Ensemble
0.8220 ±0.0040
This paper
724
MLP
0.7768
#ModelAccuracy ↑Paper
1
CNA
0.9575 ±0.0058
2
IS-GIB
0.9390 ±0.0187
3
EERM
0.9112 ±0.0145
35
GAT
actual #35 vs paper #463 428 places worse
0.8080 ±0.0026
best reported
actual #35 vs paper #463 428 places worse
paper 0.7190 ±0.0060
paper 0.7190 ±0.0060 Δ at pub 0.0670 Δ today 0.0890
44
GCN
actual #44 vs paper #561 517 places worse
0.8033 ±0.0088
best reported
actual #44 vs paper #561 517 places worse
paper 0.7090 ±0.0070
61
GraphSAGE
actual #61 vs paper #713 652 places worse
0.7923 ±0.0053
best reported
actual #61 vs paper #713 652 places worse
paper 0.6720 ±0.0090
paper 0.6720 ±0.0090 Δ at pub 0.0810 Δ today 0.1203
259
SAGE
0.7361 ±0.0190
350
MLP
0.7289 ±0.0010
407
Ensemble
actual #407 vs paper #570 163 places worse
0.7243 ±0.0026
best reported
actual #407 vs paper #570 163 places worse
paper 0.7080 ±0.0050
paper 0.7080 ±0.0050 Δ today 0.0163
414
AutoGL
0.7240 ±0.0060
This paper
416
TPE (50 trials)
0.7240 ±0.0060
This paper
511
random (10 trials)
0.7150 ±0.0060
This paper
566
None
0.7090 ±0.0070
This paper
#ModelAccuracy ↑Paper
1
LGTL
0.9518 ±0.0011
2
MixHop (LO)
0.9513 ±0.0038
3
GraphSAGE (LO)
0.9511 ±0.0036
18
GCN
0.9296 ±0.0015
125
GraphSAGE
actual #125 vs paper #745 620 places worse
0.8864 ±0.0030
best reported
actual #125 vs paper #745 620 places worse
paper 0.7680 ±0.0060
paper 0.7680 ±0.0060 Δ at pub 0.1030 Δ today 0.1184
210
MLP
0.8690 ±0.0074
546
AutoGL
0.7930 ±0.0040
This paper
638
Ensemble
0.7850 ±0.0040
This paper

TU Dortmund

#ModelAccuracy ↑Paper
1
MSH-GNN
0.9910 ±0.0030
2
ID-GNN+BEC
0.9830 ±0.0360
3
SupCosine
0.9830 ±0.0250
39
GIN
actual #39 vs paper #534 495 places worse
0.9330 ±0.0290
best reported
actual #39 vs paper #534 495 places worse
paper 0.8270 ±0.0690
paper 0.8270 ±0.0690 Δ at pub 0.0936 Δ today 0.1060
63
GCN
0.9220 ±0.0440
358
AutoGL
0.8760 ±0.0600
This paper
403
MLP
0.8660 ±0.0495
497
TPE
0.8390 ±0.1010
This paper
535
Random
0.8270 ±0.0680
This paper
595
Top-K Pooling
0.8080 ±0.0710
This paper
#ModelAccuracy ↑Paper
1
MSH-GNN
0.9410 ±0.0330
2
GRDL
0.8260 ±0.0120
3
GMN
0.8225
32
GIN
actual #32 vs paper #476 444 places worse
0.7880 ±0.0410
best reported
actual #32 vs paper #476 444 places worse
paper 0.6650 ±0.0390
67
SAGE
0.7726 ±0.0228
149
MLP
0.7600 ±0.0320
351
AutoGL
0.7330 ±0.0440
This paper
462
Top-K Pooling
0.6950 ±0.0440
This paper
SOTA performance (not significantly different from the best) Proposed method (this paper) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green) Best today — OK at publication; today's leaderboard best is elsewhere