ICLR 2023

Revisiting Link Prediction: a data perspective

Haitao Mao, Juanhui Li, Harry Shomer, Bingheng Li, Wenqi Fan, Yao Ma, Tong Zhao, Neil Shah, Jiliang Tang

Main figure for Revisiting Link Prediction: a data perspective

Abstract

Link prediction, a fundamental task on graphs, has proven indispensable in various applications, e.g., friend recommendation, protein analysis, and drug interaction prediction. However, since datasets span a multitude of domains, they could have distinct underlying mechanisms of link formation. Evidence in existing literature underscores the absence of a universally best algorithm suitable for all datasets. In this paper, we endeavor to explore principles of link prediction across diverse datasets from a data-centric perspective. We recognize three fundamental factors critical to link prediction: local structural proximity, global structural proximity, and feature proximity. We then unearth relationships among those factors where (i) global structural proximity only shows effectiveness when local structural proximity is deficient. (ii) The incompatibility can be found between feature and structural proximity. Such incompatibility leads to GNNs for Link Prediction (GNN4LP) consistently underperforming on edges where the feature proximity factor dominates. Inspired by these new insights from a data perspective, we offer practical instruction for GNN4LP model design and guidelines for selecting appropriate benchmark datasets for more comprehensive evaluations.

Proposed method: CN

Benchmark Results

Tested on 2 datasets 4 baseline values reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

OGB

#ModelHits@20 ↑Paper
1
ELGNN
0.9777 ±0.0037
2
GIDN
0.9542
3
AGDN
0.9538 ±0.0094
12
NCNC
actual #12 vs paper #19 7 places worse
0.8411 ±0.0367
best reported
actual #12 vs paper #19 7 places worse
paper 0.7023 ±0.1211
best today: PageRank Bandits for Link Prediction (NeurIPS 2024)

Difference within reported uncertainty; not counted as underperformance.

paper 0.7023 ±0.1211 Δ today 0.1388
16
BUDDY
actual #16 vs paper #60 44 places worse
0.7851 ±0.0136
best reported
actual #16 vs paper #60 44 places worse
paper 0.2960 ±0.0475
paper 0.2960 ±0.0475 Δ at pub 0.4891 Δ today 0.4891
20
SAGE
actual #20 vs paper #42 22 places worse
0.6719 ±0.0118
best reported
actual #20 vs paper #42 22 places worse
paper 0.4984 ±0.1556

Difference within reported uncertainty; not counted as underperformance.

paper 0.4984 ±0.1556 Δ at pub 0.0406 Δ today 0.1735
24
GCN
actual #24 vs paper #42 18 places worse
0.4990 ±0.0723
actual #24 vs paper #42 18 places worse
paper 0.4990 ±0.0723

Improvement within reported uncertainty; not highlighted.

paper 0.4990 ±0.0723 Δ at pub 0.0530 Δ today 0.1486
38
GraphSAGE
0.5390 ±0.0474
47
RA
actual #47 vs paper #91 44 places worse
0.4401
best reported
actual #47 vs paper #91 44 places worse
paper 0.0623
paper 0.0623 Δ at pub 0.2137 Δ today 0.3778
50
AA
actual #50 vs paper #75 25 places worse
0.3975
best reported
actual #50 vs paper #75 25 places worse
paper 0.1861
51
CN
actual #51 vs paper #78 27 places worse
0.3869
best reported
actual #51 vs paper #78 27 places worse
paper 0.1773
52
Katz
actual #52 vs paper #78 26 places worse
0.3869
best reported
actual #52 vs paper #78 26 places worse
paper 0.1773
paper 0.1773 Δ today 0.2096
73
NeoGNN
0.2095 ±0.0603
This paper
90
PPR
0.0719
This paper
93
FH
0.0474
This paper
#ModelHits@100 ↑Paper
1
OGB Leader
0.6524 ±0.0150
2
PRB
0.6347 ±0.0175
3
LPFormer
0.6332 ±0.0063
4
NCNC
0.6264 ±0.0079
This paper
9
RA
actual #9 vs paper #16 7 places worse
0.5324
best reported
actual #9 vs paper #16 7 places worse
paper 0.4933
paper 0.4933 Δ at pub 0.0391
14
SAGE
actual #14 vs paper #32 18 places worse
0.5013 ±0.0055
best reported
actual #14 vs paper #32 18 places worse
paper 0.4102 ±0.0194
paper 0.4102 ±0.0194 Δ at pub 0.0911
15
BUDDY
actual #15 vs paper #21 6 places worse
0.4985 ±0.0020
best reported
actual #15 vs paper #21 6 places worse
paper 0.4733 ±0.0196

Difference within reported uncertainty; not counted as underperformance.

paper 0.4733 ±0.0196 Δ at pub 0.0252 Δ today 0.0252
20
NeoGNN
0.4845 ±0.0101
This paper
23
MLP
actual #23 vs paper #28 5 places worse
0.4600
best reported
actual #23 vs paper #28 5 places worse
paper 0.4500 ±0.0400
best at pub: OGB (NeurIPS 2020)

Difference within reported uncertainty; not counted as underperformance.

paper 0.4500 ±0.0400 Δ at pub 0.0100 Δ today 0.0100
39
AA
0.3245
This paper
45
GCN
actual #45 vs paper #45
0.2957 ±0.0290
actual #45 vs paper #45
paper 0.2957 ±0.0290
paper 0.2957 ±0.0290 Δ at pub 0.1087
49
CN
0.2765
This paper
51
Katz
0.2765
This paper
63
GraphSAGE
0.1655 ±0.0240
71
PPR
0.0497
This paper
SOTA performance (not significantly different from the best) Proposed method (this paper) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green) Best at pub — reported value improved over the best known when published Best today — OK at publication; today's leaderboard best is elsewhere