Reinforcement learning on graphs: A survey

Mingshuo Nie, Dongming Chen, Dongqi Wang

Main figure for Reinforcement learning on graphs: A survey

Abstract

Graph mining tasks arise from many different application domains, ranging from social networks, transportation to E-commerce, etc., which have been receiving great attention from the theoretical and algorithmic design communities in recent years, and there has been some pioneering work employing the research-rich Reinforcement Learning (RL) techniques to address graph data mining tasks. However, these graph mining methods and RL models are dispersed in different research areas, which makes it hard to compare them. In this survey, we provide a comprehensive overview of RL and graph mining methods and generalize these methods to Graph Reinforcement Learning (GRL) as a unified formulation. We further discuss the applications of GRL methods across various domains and summarize the method descriptions, open-source codes, and benchmark datasets of GRL methods. Furthermore, we propose important directions and challenges to be solved in the future. As far as we know, this is the latest work on a comprehensive survey of GRL, this work provides a global view and a learning resource for scholars. In addition, we create an online open-source for both interested scholars who want to enter this rapidly developing domain and experts who would like to compare GRL methods.

Benchmark Results

Tested on 3 datasets 9 baseline values reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

Classic

#ModelAccuracy ↑Paper
1
GraphSAGE+CNA
0.9418 ±0.0033
2
GCN+CNA
0.9366 ±0.0048
3
TransformerConv
0.9350 ±0.0058
6
Policy-GNN
0.9190 ±0.0140
This paper
21
GCN
actual #21 vs paper #573 552 places worse
0.9004
best reported
actual #21 vs paper #573 552 places worse
paper 0.8150
paper 0.8150 Δ at pub 0.0567 Δ today 0.0854
39
SAGE
0.8911 ±0.0012
44
GAT
actual #44 vs paper #424 380 places worse
0.8897 ±0.0014
best reported
actual #44 vs paper #424 380 places worse
paper 0.8300 ±0.0070
paper 0.8300 ±0.0070 Δ at pub 0.0590 Δ today 0.0597
177
DGI
actual #177 vs paper #486 309 places worse
0.8630 ±0.0020
best reported
actual #177 vs paper #486 309 places worse
paper 0.8230 ±0.0060
346
AGNN
0.8360 ±0.0030
This paper
391
LGCN
0.8330 ±0.0050
This paper
724
MLP
0.7768
#ModelAccuracy ↑Paper
1
CNA
0.9575 ±0.0058
2
IS-GIB
0.9390 ±0.0187
3
EERM
0.9112 ±0.0145
9
Policy-GNN
0.8970 ±0.0210
This paper
35
GAT
actual #35 vs paper #388 353 places worse
0.8080 ±0.0026
best reported
actual #35 vs paper #388 353 places worse
paper 0.7250 ±0.0070
paper 0.7250 ±0.0070 Δ at pub 0.0763 Δ today 0.0830
44
GCN
actual #44 vs paper #603 559 places worse
0.8033 ±0.0088
best reported
actual #44 vs paper #603 559 places worse
paper 0.7030
61
GraphSAGE
0.7923 ±0.0053
66
DGI
actual #66 vs paper #474 408 places worse
0.7890 ±0.0020
best reported
actual #66 vs paper #474 408 places worse
paper 0.7180 ±0.0070
242
AGNN
actual #242 vs paper #242
0.7380 ±0.0070
actual #242 vs paper #242
paper 0.7380 ±0.0070

Improvement within reported uncertainty; not highlighted.

paper 0.7380 ±0.0070 Δ at pub 0.0210
318
GraphNAS
0.7310 ±0.0090
This paper
327
LGCN
0.7300 ±0.0060
This paper
350
MLP
0.7289 ±0.0010
#ModelAccuracy ↑Paper
1
LGTL
0.9518 ±0.0011
2
MixHop (LO)
0.9513 ±0.0038
3
GraphSAGE (LO)
0.9511 ±0.0036
18
GCN
actual #18 vs paper #585 567 places worse
0.9296 ±0.0015
best reported
actual #18 vs paper #585 567 places worse
paper 0.7900
20
GAT
actual #20 vs paper #585 565 places worse
0.9233 ±0.0018
best reported
actual #20 vs paper #585 565 places worse
paper 0.7900 ±0.0030
21
Policy-GNN
0.9210 ±0.0220
This paper
116
SAGE
0.8881 ±0.0015
204
DGI
actual #204 vs paper #745 541 places worse
0.8708 ±0.0017
best reported
actual #204 vs paper #745 541 places worse
paper 0.7680 ±0.0060
210
MLP
0.8690 ±0.0074
465
AGNN
actual #465 vs paper #482 17 places worse
0.7990 ±0.0700
best reported
actual #465 vs paper #482 17 places worse
paper 0.7970 ±0.0040

Difference within reported uncertainty; not counted as underperformance.

paper 0.7970 ±0.0040 Δ at pub 0.0020
493
GraphNAS
0.7960 ±0.0040
This paper
505
LGCN
0.7950 ±0.0020
This paper
SOTA performance (not significantly different from the best) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green)