ICLR 2021

16 Pre-training Molecular Graph Representation with 3D Geometry

Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, Jian Tang

Main figure for 16 Pre-training Molecular Graph Representation with 3D Geometry

Abstract

Molecular graph representation learning is a fundamental problem in modern drug and material discovery. Molecular graphs are typically modeled by their 2D topological structures, but it has been recently discovered that 3D geometric information plays a more vital role in predicting molecular functionalities. However, the lack of 3D information in real-world scenarios has significantly impeded the learning of geometric graph representation. To cope with this challenge, we propose the Graph Multi-View Pre-training (GraphMVP) framework where self-supervised learning (SSL) is performed by leveraging the correspondence and consistency between 2D topological structures and 3D geometric views. GraphMVP effectively learns a 2D molecular graph encoder that is enhanced by richer and more discriminative 3D geometry. We further provide theoretical insights to justify the effectiveness of GraphMVP. Finally, comprehensive experiments show that GraphMVP can consistently outperform existing graph SSL methods. Code is available on GitHub.

Benchmark Results

Tested on 7 datasets 5 baseline values reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

MoleculeNet

#ModelROC-AUC ↑Paper
1
HIMP
0.8740 ±0.0050
2
GINE w/ VN
0.8720 ±0.0030
3
NaiveGINe+ K=3 w/ VN
0.8700 ±0.0040
14
GCN
0.8400 ±0.0040
31
JOAO
actual #31 vs paper #247 216 places worse
0.8230 ±0.0220
best reported
actual #31 vs paper #247 216 places worse
paper 0.7440 ±0.0070
paper 0.7440 ±0.0070 Δ today 0.0790
35
EdgePred
actual #35 vs paper #246 211 places worse
0.8180 ±0.0250
best reported
actual #35 vs paper #246 211 places worse
paper 0.7450 ±0.0040
paper 0.7450 ±0.0040 Δ today 0.0730
46
ContextPred
actual #46 vs paper #279 233 places worse
0.8060 ±0.0120
best reported
actual #46 vs paper #279 233 places worse
paper 0.7330 ±0.0050
paper 0.7330 ±0.0050 Δ today 0.0730
47
GraphCL
actual #47 vs paper #231 184 places worse
0.8050 ±0.0170
best reported
actual #47 vs paper #231 184 places worse
paper 0.7500 ±0.0030
paper 0.7500 ±0.0030 Δ today 0.0550
61
GraphMVP
actual #61 vs paper #246 185 places worse
0.7990 ±0.0180
best reported
actual #61 vs paper #246 185 places worse
paper 0.7450 ±0.0040
paper 0.7450 ±0.0040 Δ today 0.0540
71
GraphLoG
actual #71 vs paper #282 211 places worse
0.7960 ±0.0250
best reported
actual #71 vs paper #282 211 places worse
paper 0.7300 ±0.0030
paper 0.7300 ±0.0030 Δ today 0.0660
81
InfoGraph
actual #81 vs paper #282 201 places worse
0.7930
best reported
actual #81 vs paper #282 201 places worse
paper 0.7300 ±0.0070
paper 0.7300 ±0.0070 Δ at pub 0.0630
180
GPT-GNN
actual #180 vs paper #211 31 places worse
0.7610 ±0.0040
best reported
actual #180 vs paper #211 31 places worse
paper 0.7530 ±0.0050

Difference within reported uncertainty; not counted as underperformance.

paper 0.7530 ±0.0050 Δ at pub 0.0080
184
GraphMVP-G
0.7590 ±0.0050
This paper
220
G-Contextual
0.7520 ±0.0030
This paper
229
AttrMask
actual #229 vs paper #259 30 places worse
0.7510 ±0.0090
best reported
actual #229 vs paper #259 30 places worse
paper 0.7420 ±0.0080

Difference within reported uncertainty; not counted as underperformance.

paper 0.7420 ±0.0080 Δ today 0.0090
236
--
0.7490 ±0.0080
This paper
249
GraphMVP-C
0.7440 ±0.0020
This paper
280
G-Motif
0.7320 ±0.0080
This paper
#ModelROC-AUC ↑Paper
1
ProtoMol
0.9140 ±0.0030
2
MMSG
0.9080 ±0.0050
3
MemGNN
0.9070
43
GIN
0.8646 ±0.0081
98
GraphLoG
actual #98 vs paper #206 108 places worse
0.8350 ±0.0120
best reported
actual #98 vs paper #206 108 places worse
paper 0.7880 ±0.0070
paper 0.7880 ±0.0070 Δ today 0.0470
100
JOAO
actual #100 vs paper #261 161 places worse
0.8340 ±0.0330
best reported
actual #100 vs paper #261 161 places worse
paper 0.7290 ±0.0200
paper 0.7290 ±0.0200 Δ today 0.1050
149
GraphMVP
actual #149 vs paper #232 83 places worse
0.8120 ±0.0090
best reported
actual #149 vs paper #232 83 places worse
paper 0.7680 ±0.0110
paper 0.7680 ±0.0110 Δ today 0.0440
150
GraphMVP-C
0.8120 ±0.0090
This paper
176
EdgePred
actual #176 vs paper #295 119 places worse
0.7990 ±0.0090
best reported
actual #176 vs paper #295 119 places worse
paper 0.6460 ±0.0470
paper 0.6460 ±0.0470 Δ today 0.1530
182
AttrMask
actual #182 vs paper #226 44 places worse
0.7970 ±0.0033
best reported
actual #182 vs paper #226 44 places worse
paper 0.7720 ±0.0140

Difference within reported uncertainty; not counted as underperformance.

paper 0.7720 ±0.0140 Δ today 0.0250
186
ContextPred
actual #186 vs paper #207 21 places worse
0.7960 ±0.0120
best reported
actual #186 vs paper #207 21 places worse
paper 0.7860 ±0.0140

Difference within reported uncertainty; not counted as underperformance.

paper 0.7860 ±0.0140 Δ today 0.0100
192
GraphMVP-G
0.7930 ±0.0150
This paper
196
G-Contextual
0.7920 ±0.0030
This paper
204
InfoGraph
actual #204 vs paper #255 51 places worse
0.7890 ±0.0110
best reported
actual #204 vs paper #255 51 places worse
paper 0.7390 ±0.0250

Difference within reported uncertainty; not counted as underperformance.

paper 0.7390 ±0.0250 Δ today 0.0500
208
GPT-GNN
actual #208 vs paper #225 17 places worse
0.7860 ±0.0290
best reported
actual #208 vs paper #225 17 places worse
paper 0.7760 ±0.0050

Difference within reported uncertainty; not counted as underperformance.

paper 0.7760 ±0.0050 Δ at pub 0.0090 Δ today 0.0100
213
GraphCL
actual #213 vs paper #286 73 places worse
0.7820 ±0.0120
best reported
actual #213 vs paper #286 73 places worse
paper 0.6870 ±0.0780

Difference within reported uncertainty; not counted as underperformance.

paper 0.6870 ±0.0780 Δ today 0.0950
259
G-Motif
0.7340 ±0.0400
This paper
263
--
0.7260 ±0.0490
This paper
#ModelROC-AUC ↑Paper
1
MAGPrompt+
0.9432 ±0.0238
2
CamS-LLaMA
0.9420
3
SE(3)-I MPPH
0.9400 ±0.0210
34
GIN
0.8875 ±0.0049
36
GraphCL
actual #36 vs paper #219 183 places worse
0.8870 ±0.0190
best reported
actual #36 vs paper #219 183 places worse
paper 0.6750 ±0.0330
paper 0.6750 ±0.0330 Δ today 0.2120
39
JOAO
actual #39 vs paper #253 214 places worse
0.8790 ±0.0200
best reported
actual #39 vs paper #253 214 places worse
paper 0.6600 ±0.0060
paper 0.6600 ±0.0060 Δ today 0.2190
40
ContextPred
actual #40 vs paper #118 78 places worse
0.8770 ±0.0260
best reported
actual #40 vs paper #118 78 places worse
paper 0.7120 ±0.0090
paper 0.7120 ±0.0090 Δ today 0.1650
44
EdgePred
actual #44 vs paper #278 234 places worse
0.8730 ±0.0160
best reported
actual #44 vs paper #278 234 places worse
paper 0.6450 ±0.0310
paper 0.6450 ±0.0310 Δ today 0.2280
49
GraphMVP
actual #49 vs paper #190 141 places worse
0.8600 ±0.0340
best reported
actual #49 vs paper #190 141 places worse
paper 0.6850 ±0.0020
paper 0.6850 ±0.0020 Δ today 0.1750
54
GraphLoG
actual #54 vs paper #205 151 places worse
0.8460 ±0.0080
best reported
actual #54 vs paper #205 151 places worse
paper 0.6780 ±0.0170
paper 0.6780 ±0.0170 Δ today 0.1680
94
GraphMVP-C
0.7240 ±0.0160
This paper
124
GraphMVP-G
0.7080 ±0.0050
This paper
135
G-Contextual
0.7030 ±0.0160
This paper
136
G-Motif
actual #136 vs paper #245 109 places worse
0.7030 ±0.0150
best reported
actual #136 vs paper #245 109 places worse
paper 0.6640 ±0.0340

Difference within reported uncertainty; not counted as underperformance.

paper 0.6640 ±0.0340 Δ today 0.0390
141
AttrMask
0.7020 ±0.0050
This paper
173
InfoGraph
0.6920 ±0.0080
This paper
220
GPT-GNN
actual #220 vs paper #278 58 places worse
0.6750 ±0.0130
best reported
actual #220 vs paper #278 58 places worse
paper 0.6450 ±0.0110
paper 0.6450 ±0.0110 Δ at pub 0.0300
268
--
0.6540 ±0.0240
This paper
#ModelROC-AUC ↑Paper
1
SE(3)-I MPPH
0.9930 ±0.0040
2
KA-GCN
0.9920 ±0.0050
3
KA-GAT
0.9910 ±0.0050
19
GCN
0.9130 ±0.0170
40
AttrMask
actual #40 vs paper #183 143 places worse
0.8770 ±0.0119
best reported
actual #40 vs paper #183 143 places worse
paper 0.6860 ±0.0960

Difference within reported uncertainty; not counted as underperformance.

paper 0.6860 ±0.0960 Δ today 0.1910
49
GraphMVP
actual #49 vs paper #96 47 places worse
0.8650 ±0.0170
best reported
actual #49 vs paper #96 47 places worse
paper 0.7900 ±0.0250
paper 0.7900 ±0.0250 Δ today 0.0750
78
G-Motif
actual #78 vs paper #106 28 places worse
0.8140 ±0.0220
best reported
actual #78 vs paper #106 28 places worse
paper 0.7780 ±0.0200

Difference within reported uncertainty; not counted as underperformance.

paper 0.7780 ±0.0200 Δ today 0.0360
79
JOAO
actual #79 vs paper #195 116 places worse
0.8132 ±0.0249
best reported
actual #79 vs paper #195 116 places worse
paper 0.6630 ±0.0390
paper 0.6630 ±0.0390 Δ today 0.1502
96
GraphMVP-G
0.7910 ±0.0280
This paper
97
GraphCL
0.7890 ±0.0420
This paper
103
GraphLoG
actual #103 vs paper #216 113 places worse
0.7872 ±0.0258
best reported
actual #103 vs paper #216 113 places worse
paper 0.6200 ±0.0180
paper 0.6200 ±0.0180 Δ today 0.1672
106
InfoGraph
actual #106 vs paper #128 22 places worse
0.7810
best reported
actual #106 vs paper #128 22 places worse
paper 0.7510 ±0.0500

Difference within reported uncertainty; not counted as underperformance.

paper 0.7510 ±0.0500 Δ at pub 0.0300
111
GraphMVP-C
0.7750 ±0.0420
This paper
115
ContextPred
actual #115 vs paper #142 27 places worse
0.7370 ±0.0400
actual #115 vs paper #142 27 places worse
paper 0.7370 ±0.0400
131
GPT-GNN
actual #131 vs paper #232 101 places worse
0.7490 ±0.0270
best reported
actual #131 vs paper #232 101 places worse
paper 0.5780 ±0.0310
paper 0.5780 ±0.0310 Δ at pub 0.1710
191
G-Contextual
actual #191 vs paper #222 31 places worse
0.6760 ±0.0290
best reported
actual #191 vs paper #222 31 places worse
paper 0.5990 ±0.0820

Difference within reported uncertainty; not counted as underperformance.

paper 0.5990 ±0.0820 Δ today 0.0770
192
EdgePred
actual #192 vs paper #237 45 places worse
0.5580 ±0.0620
actual #192 vs paper #237 45 places worse
paper 0.5580 ±0.0620

Improvement within reported uncertainty; not highlighted.

paper 0.5580 ±0.0620 Δ at pub 0.0540
226
--
0.5880 ±0.0550
This paper
#ModelROC-AUC ↑Paper
1
KA-GAT
0.8470 ±0.0020
2
KA-GCN
0.8420 ±0.0010
3
GraphKAN
0.8370 ±0.0010
66
GIN
0.6403 ±0.0104
68
JOAO
actual #68 vs paper #150 82 places worse
0.6400 ±0.0100
best reported
actual #68 vs paper #150 82 places worse
paper 0.6070 ±0.0100
paper 0.6070 ±0.0100 Δ today 0.0330
70
GraphMVP
actual #70 vs paper #99 29 places worse
0.6390 ±0.0120
best reported
actual #70 vs paper #99 29 places worse
paper 0.6230 ±0.0160

Difference within reported uncertainty; not counted as underperformance.

paper 0.6230 ±0.0160 Δ today 0.0160
71
GraphMVP-C
0.6390 ±0.0120
This paper
103
ContextPred
actual #103 vs paper #196 93 places worse
0.5930 ±0.0140
actual #103 vs paper #196 93 places worse
paper 0.5930 ±0.0140
114
GraphCL
actual #114 vs paper #178 64 places worse
0.6183 ±0.0060
best reported
actual #114 vs paper #178 64 places worse
paper 0.6010 ±0.0130

Difference within reported uncertainty; not counted as underperformance.

paper 0.6010 ±0.0130 Δ today 0.0173
123
GraphLoG
actual #123 vs paper #220 97 places worse
0.6150 ±0.0130
best reported
actual #123 vs paper #220 97 places worse
paper 0.5740 ±0.0230

Difference within reported uncertainty; not counted as underperformance.

paper 0.5740 ±0.0230 Δ today 0.0410
134
AttrMask
actual #134 vs paper #169 35 places worse
0.6120 ±0.0012
best reported
actual #134 vs paper #169 35 places worse
paper 0.6040 ±0.0060

Difference within reported uncertainty; not counted as underperformance.

paper 0.6040 ±0.0060 Δ today 0.0080
139
G-Contextual
actual #139 vs paper #206 67 places worse
0.6110 ±0.0250
best reported
actual #139 vs paper #206 67 places worse
paper 0.5840 ±0.0060

Difference within reported uncertainty; not counted as underperformance.

paper 0.5840 ±0.0060 Δ today 0.0270
158
G-Motif
0.6060 ±0.0110
This paper
162
InfoGraph
actual #162 vs paper #199 37 places worse
0.5920 ±0.0020
actual #162 vs paper #199 37 places worse
paper 0.5920 ±0.0020
paper 0.5920 ±0.0020 Δ at pub 0.0070
170
EdgePred
actual #170 vs paper #226 56 places worse
0.5670 ±0.0010
actual #170 vs paper #226 56 places worse
paper 0.5670 ±0.0010
paper 0.5670 ±0.0010 Δ at pub 0.0650 Δ today 0.0370
177
GraphMVP-G
0.6020 ±0.0110
This paper
197
GPT-GNN
actual #197 vs paper #219 22 places worse
0.5930 ±0.0080
best reported
actual #197 vs paper #219 22 places worse
paper 0.5750 ±0.0420

Difference within reported uncertainty; not counted as underperformance.

paper 0.5750 ±0.0420 Δ at pub 0.0180
214
--
0.5800 ±0.0240
This paper
#ModelROC-AUC ↑Paper
1
GeomGCL
0.7630
2
LiGhT DeepGPT
0.7570 ±0.0110
3
CoMPT
0.7550
14
GCN
0.7350 ±0.0020
28
ContextPred
actual #28 vs paper #167 139 places worse
0.7140 ±0.0180
best reported
actual #28 vs paper #167 139 places worse
paper 0.6280 ±0.0030
paper 0.6280 ±0.0030 Δ today 0.0860
30
EdgePred
actual #30 vs paper #194 164 places worse
0.7120 ±0.0110
best reported
actual #30 vs paper #194 164 places worse
paper 0.6080 ±0.0050
paper 0.6080 ±0.0050 Δ today 0.1040
31
JOAO
actual #31 vs paper #168 137 places worse
0.7110 ±0.0140
best reported
actual #31 vs paper #168 137 places worse
paper 0.6270 ±0.0060
paper 0.6270 ±0.0060 Δ today 0.0840
35
InfoGraph
actual #35 vs paper #184 149 places worse
0.7050
best reported
actual #35 vs paper #184 149 places worse
paper 0.6200 ±0.0030
paper 0.6200 ±0.0030 Δ at pub 0.0850
42
GraphCL
actual #42 vs paper #167 125 places worse
0.6960 ±0.0230
best reported
actual #42 vs paper #167 125 places worse
paper 0.6280 ±0.0020
paper 0.6280 ±0.0020 Δ today 0.0680
50
GraphMVP
actual #50 vs paper #168 118 places worse
0.6890 ±0.0100
best reported
actual #50 vs paper #168 118 places worse
paper 0.6270 ±0.0010
paper 0.6270 ±0.0010 Δ today 0.0620
64
GraphLoG
actual #64 vs paper #181 117 places worse
0.6770 ±0.0190
best reported
actual #64 vs paper #181 117 places worse
paper 0.6220 ±0.0040
paper 0.6220 ±0.0040 Δ today 0.0550
150
AttrMask
actual #150 vs paper #177 27 places worse
0.6330 ±0.0060
best reported
actual #150 vs paper #177 27 places worse
paper 0.6250 ±0.0040

Difference within reported uncertainty; not counted as underperformance.

paper 0.6250 ±0.0040 Δ today 0.0080
151
GPT-GNN
actual #151 vs paper #181 30 places worse
0.6330 ±0.0030
best reported
actual #151 vs paper #181 30 places worse
paper 0.6220 ±0.0010
paper 0.6220 ±0.0010 Δ at pub 0.0090 Δ today 0.0110
161
GraphMVP-G
0.6310 ±0.0020
This paper
162
GraphMVP-C
0.6310 ±0.0040
This paper
172
G-Contextual
0.6260 ±0.0030
This paper
173
G-Motif
0.6260 ±0.0050
This paper
186
--
0.6160 ±0.0120
This paper
#ModelRMSE ↓Paper
1
Graphormer-SPIS
0.4840 ±0.0050
2
Graphormer-SPD
0.4920 ±0.0040
3
conan-FGW
0.5140 ±0.0190
7
GCN
0.5640 ±0.0150
89
GraphMVP-C
1.0290
This paper
92
GraphMVP-G
1.0640
This paper
98
GraphMVP
1.0910
This paper
104
AM
1.1120
This paper
105
JOAO
1.1200
This paper
112
--
1.1780
This paper
115
CP
1.1960
This paper
SOTA performance (not significantly different from the best) Proposed method (this paper) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green) Best at pub — reported value improved over the best known when published Improved at pub — later paper surpassed this result Best today — OK at publication; today's leaderboard best is elsewhere