Pairwise Learning for Neural Link Prediction

Zhitao Wang, Yong Zhou, L. Hong, Yuanhang Zou, Hanjing Su, Shouzhi Chen

Main figure for Pairwise Learning for Neural Link Prediction

Abstract

In this paper, we aim at providing an effective Pairwise Learning for Neural Link Prediction (PLNLP) framework. The framework treats link prediction as a pairwise learning to rank problem and consists of four main components, i.e., neighborhood encoder, link predictor, negative sampler and objective function. The framework is flexible that any generic graph neural convolutions or link prediction specific neural architectures could be employed as neighborhood encoder. For link predictor, we design different scoring functions, which could be selected based on different types of graphs. In negative sampler, we provide several sampling strategies, which are problem specific. As for objective function, we propose to use an effective ranking loss, which approximately maximizes the standard ranking metric AUC. We evaluate the proposed PLNLP framework on 4 link property prediction datasets of Open Graph Benchmark (OGB), including ogbl-ddi, ogbl-collab, ogbl-ppa and ogbl-ciation2. PLNLP achieves top 1 performance on ogbl-ddi and ogbl-collab, and top 2 performance on ogbl-ciation2 only with basic neural architecture. The experimental results demonstrate the effectiveness of PLNLP.

Proposed method: PLNLP

Benchmark Results

Tested on 4 datasets 1 baseline value reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

OGB

#ModelHits@50 ↑Paper
1
PROXI
0.7650 ±0.0027
2
OGB Leader
0.7129 ±0.0018
3
GIDN
0.7096 ±0.0055
4
PLNLP
0.7059 ±0.0029
This paper
5
HOP-Rec
0.7012 ±0.0016
This paper
9
GCN
actual #9 vs paper #85 76 places worse
0.6621 ±0.0033
best reported
actual #9 vs paper #85 76 places worse
paper 0.4714 ±0.0145
best at pub: OGB (NeurIPS 2020)
paper 0.4714 ±0.0145 Δ today 0.1907
16
AA+Proposal Set
0.6548
This paper
20
SEAL
0.6474 ±0.0033
This paper
23
AA
actual #23 vs paper #24 1 place worse
0.6435
best reported
actual #23 vs paper #24 1 place worse
paper 0.6417
27
CN
0.6137
This paper
28
SAGE
actual #28 vs paper #40 12 places worse
0.5944 ±0.0137
best reported
actual #28 vs paper #40 12 places worse
paper 0.5463 ±0.0112
paper 0.5463 ±0.0112 Δ today 0.0481
41
GraphSAGE
0.5463 ±0.0112
60
DeepWalk
actual #60 vs paper #60
0.5040 ±0.0030
best reported
actual #60 vs paper #60
paper 0.5037 ±0.0034

Difference within reported uncertainty; not counted as underperformance.

paper 0.5037 ±0.0034 Δ at pub 0.0003
68
Node2Vec
actual #68 vs paper #76 8 places worse
0.4929 ±0.0064
best reported
actual #68 vs paper #76 8 places worse
paper 0.4888 ±0.0054

Difference within reported uncertainty; not counted as underperformance.

paper 0.4888 ±0.0054 Δ at pub 0.0002 Δ today 0.0041
94
MLP
0.4438 ±0.0347
98
MF
actual #98 vs paper #101 3 places worse
0.4181 ±0.0167
best reported
actual #98 vs paper #101 3 places worse
paper 0.3886 ±0.0029
paper 0.3886 ±0.0029 Δ at pub 0.0004 Δ today 0.0295
#ModelHits@20 ↑Paper
1
ELGNN
0.9777 ±0.0037
2
GIDN
0.9542
3
AGDN
0.9538 ±0.0094
5
PLNLP
0.9088 ±0.0313
This paper
8
SAGE+Edge Attr
0.8781 ±0.0447
This paper
10
CFLP (w/ JKNet)
0.8608 ±0.0198
This paper
19
SAGE+Proposal Set
0.7495 ±0.0317
This paper
20
SAGE
actual #20 vs paper #37 17 places worse
0.6719 ±0.0118
best reported
actual #20 vs paper #37 17 places worse
paper 0.5390 ±0.0474
paper 0.5390 ±0.0474 Δ today 0.1329
24
GCN
actual #24 vs paper #53 29 places worse
0.6476 ±0.0145
best reported
actual #24 vs paper #53 29 places worse
paper 0.3707 ±0.0507
best at pub: OGB (NeurIPS 2020)
paper 0.3707 ±0.0507 Δ today 0.2769
25
Node2Vec
actual #25 vs paper #68 43 places worse
0.6363 ±0.0205
best reported
actual #25 vs paper #68 43 places worse
paper 0.2326 ±0.0209
paper 0.2326 ±0.0209 Δ at pub 0.0004 Δ today 0.4037
31
MF
actual #31 vs paper #81 50 places worse
0.5950 ±0.0168
best reported
actual #31 vs paper #81 50 places worse
paper 0.1368 ±0.0475
paper 0.1368 ±0.0475 Δ at pub 0.0002 Δ today 0.4582
38
GraphSAGE
0.5390 ±0.0474
43
SEAL
actual #43 vs paper #59 16 places worse
0.4974 ±0.0239
best reported
actual #43 vs paper #59 16 places worse
paper 0.3056 ±0.0386
50
AA
actual #50 vs paper #75 25 places worse
0.3975
best reported
actual #50 vs paper #75 25 places worse
paper 0.1861
51
CN
actual #51 vs paper #78 27 places worse
0.3869
best reported
actual #51 vs paper #78 27 places worse
paper 0.1773
65
DeepWalk
actual #65 vs paper #70 5 places worse
0.2640 ±0.0610
best reported
actual #65 vs paper #70 5 places worse
paper 0.2246 ±0.0290

Difference within reported uncertainty; not counted as underperformance.

paper 0.2246 ±0.0290 Δ at pub 0.0394
#ModelHits@100 ↑Paper
1
OGB Leader
0.6524 ±0.0150
2
PRB
0.6347 ±0.0175
3
LPFormer
0.6332 ±0.0063
9
RA
0.5324
This paper
14
SAGE
actual #14 vs paper #62 48 places worse
0.5013 ±0.0055
best reported
actual #14 vs paper #62 48 places worse
paper 0.1655 ±0.0240
paper 0.1655 ±0.0240 Δ today 0.3358
19
SEAL
0.4880 ±0.0316
This paper
23
MLP
0.4600
39
AA
0.3245
This paper
41
PLNLP
0.3238 ±0.0258
This paper
42
MF
actual #42 vs paper #42
0.3230 ±0.0090
best reported
actual #42 vs paper #42
paper 0.3229 ±0.0094

Difference within reported uncertainty; not counted as underperformance.

paper 0.3229 ±0.0094 Δ at pub 0.0001
45
GCN
actual #45 vs paper #59 14 places worse
0.3084 ±0.0178
best reported
actual #45 vs paper #59 14 places worse
paper 0.1867 ±0.0132
paper 0.1867 ±0.0132 Δ at pub 0.0003 Δ today 0.1217
46
DeepWalk
actual #46 vs paper #55 9 places worse
0.2890 ±0.0150
best reported
actual #46 vs paper #55 9 places worse
paper 0.2302 ±0.0163
paper 0.2302 ±0.0163 Δ at pub 0.0588
49
CN
actual #49 vs paper #49
0.2765
actual #49 vs paper #49
paper 0.2765
57
Node2Vec
actual #57 vs paper #57
0.2230 ±0.0080
best reported
actual #57 vs paper #57
paper 0.2226 ±0.0083

Difference within reported uncertainty; not counted as underperformance.

paper 0.2226 ±0.0083 Δ at pub 0.0004
64
Classification
0.1655
This paper
63
GraphSAGE
0.1655 ±0.0240
#ModelMRR ↑Paper
1
CFG
0.8997 ±0.0015
2
SIEG
0.8987 ±0.0018
3
LPFormer
0.8981 ±0.0013
8
SEAL
0.8767 ±0.0032
This paper
12
GCN
actual #12 vs paper #21 9 places worse
0.8705 ±0.0004
best reported
actual #12 vs paper #21 9 places worse
paper 0.8474 ±0.0031
best at pub: OGB (NeurIPS 2020)
paper 0.8474 ±0.0031 Δ today 0.0231
20
PLNLP
0.8492 ±0.0029
This paper
23
Classification
0.8464
This paper
30
SAGE
actual #30 vs paper #32 2 places worse
0.8292 ±0.0022
best reported
actual #30 vs paper #32 2 places worse
paper 0.8260 ±0.0036

Difference within reported uncertainty; not counted as underperformance.

paper 0.8260 ±0.0036 Δ today 0.0032
32
GraphSAGE
0.8264 ±0.0001
52
AA
actual #52 vs paper #61 9 places worse
0.7596
best reported
actual #52 vs paper #61 9 places worse
paper 0.5189
54
CN
actual #54 vs paper #63 9 places worse
0.7430
best reported
actual #54 vs paper #63 9 places worse
paper 0.5147
62
MF
0.5186 ±0.0443
This paper
71
MLP
0.3917 ±0.0044
SOTA performance (not significantly different from the best) Proposed method (this paper) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green) Best at pub — reported value improved over the best known when published Best today — OK at publication; today's leaderboard best is elsewhere