Motif-aware Attribute Masking for Molecular Graph Pre-training

Eric Inae, Gang Liu, Meng Jiang

Main figure for Motif-aware Attribute Masking for Molecular Graph Pre-training

Abstract

Attribute reconstruction is used to predict node or edge features in the pre-training of graph neural networks. Given a large number of molecules, they learn to capture structural knowledge, which is transferable for various downstream property prediction tasks and vital in chemistry, biomedicine, and material science. Previous strategies that randomly select nodes to do attribute masking leverage the information of local neighbors. However, the over-reliance of these neighbors inhibits the model's ability to learn long-range dependencies from higher-level substructures, such as functional groups or chemical motifs. To explicitly measure and encourage the inter-motif knowledge transfer in pre-trained models, we define inter-motif node influence measures and propose a novel motif-aware attribute masking strategy to capture long-range inter-motif structures by leveraging the information of atoms in neighboring motifs. Once each graph is decomposed into disjoint motifs, the features for every node within a sample motif are masked and subsequently predicted using a graph decoder. We evaluate our approach on eleven molecular classification and regression datasets and demonstrate its advantages.

Proposed method: MoAMa

Benchmark Results

Tested on 9 datasets 31 baseline values reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

MoleculeNet

#ModelROC-AUC ↑Paper
1
HIMP
0.8740 ±0.0050
2
GINE w/ VN
0.8720 ±0.0030
3
NaiveGINe+ K=3 w/ VN
0.8700 ±0.0040
14
GCN
0.8400 ±0.0040
15
GROVER
actual #15 vs paper #175 160 places worse
0.8380 ±0.0170
best reported
actual #15 vs paper #175 160 places worse
paper 0.7630 ±0.0060
paper 0.7630 ±0.0060 Δ at pub 0.0750
31
JOAO
actual #31 vs paper #238 207 places worse
0.8230 ±0.0220
best reported
actual #31 vs paper #238 207 places worse
paper 0.7480 ±0.0060
paper 0.7480 ±0.0060 Δ at pub 0.0750
46
ContextPred
actual #46 vs paper #269 223 places worse
0.8060 ±0.0120
best reported
actual #46 vs paper #269 223 places worse
paper 0.7360 ±0.0030
paper 0.7360 ±0.0030 Δ at pub 0.0700
72
Uni-Mol
actual #72 vs paper #92 20 places worse
0.7960 ±0.0050
best reported
actual #72 vs paper #92 20 places worse
paper 0.7890 ±0.0040

Difference within reported uncertainty; not counted as underperformance.

paper 0.7890 ±0.0040 Δ today 0.0070
71
GraphLoG
actual #71 vs paper #193 122 places worse
0.7960 ±0.0250
best reported
actual #71 vs paper #193 122 places worse
paper 0.7570 ±0.0050
paper 0.7570 ±0.0050 Δ at pub 0.0390
86
No pretrain
actual #86 vs paper #242 156 places worse
0.7920 ±0.0199
best reported
actual #86 vs paper #242 156 places worse
paper 0.7460 ±0.0040
paper 0.7460 ±0.0040 Δ at pub 0.0460
112
MoAMa
actual #112 vs paper #170 58 places worse
0.7829 ±0.0055
best reported
actual #112 vs paper #170 58 places worse
paper 0.7650 ±0.0010
150
Mole-BERT
actual #150 vs paper #160 10 places worse
0.7710 ±0.0040
best reported
actual #150 vs paper #160 10 places worse
paper 0.7680 ±0.0050

Difference within reported uncertainty; not counted as underperformance.

paper 0.7680 ±0.0050 Δ today 0.0030
160
D-SLA
actual #160 vs paper #160
0.7681 ±0.0052
best reported
actual #160 vs paper #160
paper 0.7680 ±0.0050

Difference within reported uncertainty; not counted as underperformance.

paper 0.7680 ±0.0050 Δ at pub 0.0001
172
MGSSL
actual #172 vs paper #219 47 places worse
0.7650 ±0.0030
best reported
actual #172 vs paper #219 47 places worse
paper 0.7520 ±0.0060
paper 0.7520 ±0.0060 Δ at pub 0.0130
202
GraphMAE
0.7550 ±0.0060
This paper
229
AttrMask
actual #229 vs paper #229
0.7510 ±0.0090
actual #229 vs paper #229
paper 0.7510 ±0.0090

Improvement within reported uncertainty; not highlighted.

paper 0.7510 ±0.0090 Δ at pub 0.0030
251
MCM
0.7440 ±0.0010
This paper
#ModelROC-AUC ↑Paper
1
ProtoMol
0.9140 ±0.0030
2
MMSG
0.9080 ±0.0050
3
MemGNN
0.9070
22
No pretrain
actual #22 vs paper #263 241 places worse
0.8743 ±0.0163
best reported
actual #22 vs paper #263 241 places worse
paper 0.7240 ±0.0380
paper 0.7240 ±0.0380 Δ at pub 0.1503
43
GIN
0.8646 ±0.0081
61
Uni-Mol
actual #61 vs paper #214 153 places worse
0.8570 ±0.0020
best reported
actual #61 vs paper #214 153 places worse
paper 0.7810 ±0.0140
paper 0.7810 ±0.0140 Δ today 0.0760
94
D-SLA
0.8380 ±0.0100
This paper
98
GraphLoG
actual #98 vs paper #98
0.8350 ±0.0120
actual #98 vs paper #98
paper 0.8350 ±0.0120

Improvement within reported uncertainty; not highlighted.

paper 0.8350 ±0.0120 Δ at pub 0.0090
100
JOAO
actual #100 vs paper #260 160 places worse
0.8340 ±0.0330
best reported
actual #100 vs paper #260 160 places worse
paper 0.7320 ±0.0160
paper 0.7320 ±0.0160 Δ at pub 0.1020
105
GraphMAE
0.8310 ±0.0090
This paper
118
GROVER
actual #118 vs paper #187 69 places worse
0.8260 ±0.0070
best reported
actual #118 vs paper #187 69 places worse
paper 0.7950 ±0.0110
130
MGSSL
actual #130 vs paper #206 76 places worse
0.8203 ±0.0379
best reported
actual #130 vs paper #206 76 places worse
paper 0.7880 ±0.0090

Difference within reported uncertainty; not counted as underperformance.

paper 0.7880 ±0.0090 Δ at pub 0.0090 Δ today 0.0323
143
Mole-BERT
actual #143 vs paper #164 21 places worse
0.8140 ±0.0100
best reported
actual #143 vs paper #164 21 places worse
paper 0.8080 ±0.0140

Difference within reported uncertainty; not counted as underperformance.

paper 0.8080 ±0.0140 Δ today 0.0060
147
MoAMa
actual #147 vs paper #173 26 places worse
0.8132 ±0.0106
best reported
actual #147 vs paper #173 26 places worse
paper 0.8010 ±0.0050

Difference within reported uncertainty; not counted as underperformance.

paper 0.8010 ±0.0050 Δ today 0.0122
182
AttrMask
actual #182 vs paper #223 41 places worse
0.7970 ±0.0033
best reported
actual #182 vs paper #223 41 places worse
paper 0.7780 ±0.0180

Difference within reported uncertainty; not counted as underperformance.

paper 0.7780 ±0.0180 Δ at pub 0.0190 Δ today 0.0190
186
ContextPred
actual #186 vs paper #206 20 places worse
0.7960 ±0.0120
best reported
actual #186 vs paper #206 20 places worse
paper 0.7880 ±0.0120

Difference within reported uncertainty; not counted as underperformance.

paper 0.7880 ±0.0120 Δ at pub 0.0080 Δ today 0.0080
188
MCM
0.7950 ±0.0130
This paper
#ModelROC-AUC ↑Paper
1
MAGPrompt+
0.9432 ±0.0238
2
CamS-LLaMA
0.9420
3
SE(3)-I MPPH
0.9400 ±0.0210
18
No pretrain
actual #18 vs paper #259 241 places worse
0.9223 ±0.0307
best reported
actual #18 vs paper #259 241 places worse
paper 0.6570 ±0.0330
paper 0.6570 ±0.0330 Δ at pub 0.2653
34
GIN
0.8875 ±0.0049
37
GROVER
actual #37 vs paper #201 164 places worse
0.8870 ±0.0060
best reported
actual #37 vs paper #201 164 places worse
paper 0.6800 ±0.0150
paper 0.6800 ±0.0150 Δ at pub 0.2070
39
JOAO
actual #39 vs paper #245 206 places worse
0.8790 ±0.0200
best reported
actual #39 vs paper #245 206 places worse
paper 0.6640 ±0.0100
paper 0.6640 ±0.0100 Δ at pub 0.2150
40
ContextPred
actual #40 vs paper #127 87 places worse
0.8770 ±0.0260
best reported
actual #40 vs paper #127 87 places worse
paper 0.7060 ±0.0150
paper 0.7060 ±0.0150 Δ at pub 0.1710
51
MoAMa
actual #51 vs paper #86 35 places worse
0.8589 ±0.0061
best reported
actual #51 vs paper #86 35 places worse
paper 0.7280 ±0.0090
54
GraphLoG
actual #54 vs paper #90 36 places worse
0.8460 ±0.0080
best reported
actual #54 vs paper #90 36 places worse
paper 0.7250 ±0.0080
paper 0.7250 ±0.0080 Δ at pub 0.1210
61
MGSSL
actual #61 vs paper #183 122 places worse
0.7952 ±0.0198
best reported
actual #61 vs paper #183 122 places worse
paper 0.6880 ±0.0090
84
Uni-Mol
actual #84 vs paper #267 183 places worse
0.7290 ±0.0060
best reported
actual #84 vs paper #267 183 places worse
paper 0.6540 ±0.0100
paper 0.6540 ±0.0100 Δ today 0.0750
89
D-SLA
0.7260 ±0.0080
This paper
96
Mole-BERT
actual #96 vs paper #106 10 places worse
0.7230 ±0.0070
best reported
actual #96 vs paper #106 10 places worse
paper 0.7190 ±0.0160

Difference within reported uncertainty; not counted as underperformance.

paper 0.7190 ±0.0160 Δ today 0.0040
101
GraphMAE
0.7200 ±0.0060
This paper
113
MCM
0.7160 ±0.0060
This paper
141
AttrMask
actual #141 vs paper #268 127 places worse
0.7020 ±0.0050
best reported
actual #141 vs paper #268 127 places worse
paper 0.6520 ±0.0140
paper 0.6520 ±0.0140 Δ at pub 0.0500 Δ today 0.0500
#ModelROC-AUC ↑Paper
1
SE(3)-I MPPH
0.9930 ±0.0040
2
KA-GCN
0.9920 ±0.0050
3
KA-GAT
0.9910 ±0.0050
17
Uni-Mol
actual #17 vs paper #79 62 places worse
0.9190 ±0.0180
best reported
actual #17 vs paper #79 62 places worse
paper 0.8120 ±0.0170
19
GCN
0.9130 ±0.0170
28
No pretrain
actual #28 vs paper #227 199 places worse
0.8970 ±0.0093
best reported
actual #28 vs paper #227 199 places worse
paper 0.5840 ±0.0640
paper 0.5840 ±0.0640 Δ at pub 0.3130
40
AttrMask
actual #40 vs paper #145 105 places worse
0.8770 ±0.0119
best reported
actual #40 vs paper #145 105 places worse
paper 0.7350 ±0.0430
56
MoAMa
0.8530 ±0.0220
This paper
71
GraphMAE
0.8230 ±0.0120
This paper
76
GROVER
actual #76 vs paper #124 48 places worse
0.8170 ±0.0250
best reported
actual #76 vs paper #124 48 places worse
paper 0.7540 ±0.0860

Difference within reported uncertainty; not counted as underperformance.

paper 0.7540 ±0.0860 Δ at pub 0.0240 Δ today 0.0630
79
JOAO
actual #79 vs paper #195 116 places worse
0.8132 ±0.0249
best reported
actual #79 vs paper #195 116 places worse
paper 0.6660 ±0.0310
paper 0.6660 ±0.0310 Δ at pub 0.1472
85
MGSSL
actual #85 vs paper #115 30 places worse
0.8070 ±0.0210
best reported
actual #85 vs paper #115 30 places worse
paper 0.7710 ±0.0450

Difference within reported uncertainty; not counted as underperformance.

paper 0.7710 ±0.0450 Δ at pub 0.0360
87
D-SLA
actual #87 vs paper #87
0.8020 ±0.0150
actual #87 vs paper #87
paper 0.8020 ±0.0150

Improvement within reported uncertainty; not highlighted.

paper 0.8020 ±0.0150 Δ at pub 0.0003
88
Mole-BERT
actual #88 vs paper #96 8 places worse
0.8010 ±0.0360
best reported
actual #88 vs paper #96 8 places worse
paper 0.7890 ±0.0300

Difference within reported uncertainty; not counted as underperformance.

paper 0.7890 ±0.0300 Δ today 0.0120
103
GraphLoG
actual #103 vs paper #116 13 places worse
0.7872 ±0.0258
best reported
actual #103 vs paper #116 13 places worse
paper 0.7670 ±0.0330

Difference within reported uncertainty; not counted as underperformance.

paper 0.7670 ±0.0330 Δ at pub 0.0202 Δ today 0.0202
115
ContextPred
actual #115 vs paper #139 24 places worse
0.7720 ±0.0088
best reported
actual #115 vs paper #139 24 places worse
paper 0.7400 ±0.0340

Difference within reported uncertainty; not counted as underperformance.

paper 0.7400 ±0.0340 Δ at pub 0.0320 Δ today 0.0320
204
MCM
0.6470 ±0.0050
This paper
#ModelROC-AUC ↑Paper
1
KA-GAT
0.8470 ±0.0020
2
KA-GCN
0.8420 ±0.0010
3
GraphKAN
0.8370 ±0.0010
37
Uni-Mol
actual #37 vs paper #216 179 places worse
0.6590 ±0.0130
best reported
actual #37 vs paper #216 179 places worse
paper 0.5770 ±0.0030
52
GROVER
actual #52 vs paper #223 171 places worse
0.6480 ±0.0060
best reported
actual #52 vs paper #223 171 places worse
paper 0.5710 ±0.0160
58
MoAMa
0.6460 ±0.0050
This paper
66
GIN
0.6403 ±0.0104
68
JOAO
actual #68 vs paper #169 101 places worse
0.6400 ±0.0100
best reported
actual #68 vs paper #169 101 places worse
paper 0.6040 ±0.0150
paper 0.6040 ±0.0150 Δ at pub 0.0360
90
Mole-BERT
0.6280 ±0.0110
This paper
102
MCM
0.6230 ±0.0090
This paper
101
No pretrain
actual #101 vs paper #212 111 places worse
0.6230 ±0.0114
best reported
actual #101 vs paper #212 111 places worse
paper 0.5820 ±0.0170
paper 0.5820 ±0.0170 Δ at pub 0.0410
103
ContextPred
actual #103 vs paper #192 89 places worse
0.6220 ±0.0059
best reported
actual #103 vs paper #192 89 places worse
paper 0.5970 ±0.0180

Difference within reported uncertainty; not counted as underperformance.

paper 0.5970 ±0.0180 Δ at pub 0.0250 Δ today 0.0250
116
MGSSL
actual #116 vs paper #119 3 places worse
0.6180 ±0.0080
best reported
actual #116 vs paper #119 3 places worse
paper 0.6160 ±0.0100

Difference within reported uncertainty; not counted as underperformance.

paper 0.6160 ±0.0100 Δ at pub 0.0020
123
GraphLoG
actual #123 vs paper #133 10 places worse
0.6150 ±0.0130
best reported
actual #123 vs paper #133 10 places worse
paper 0.6120 ±0.0110

Difference within reported uncertainty; not counted as underperformance.

paper 0.6120 ±0.0110 Δ at pub 0.0030
134
AttrMask
actual #134 vs paper #162 28 places worse
0.6120 ±0.0012
best reported
actual #134 vs paper #162 28 places worse
paper 0.6050 ±0.0090

Difference within reported uncertainty; not counted as underperformance.

paper 0.6050 ±0.0090 Δ at pub 0.0070 Δ today 0.0070
175
GraphMAE
0.6030 ±0.0110
This paper
176
D-SLA
actual #176 vs paper #176
0.6022 ±0.0113
best reported
actual #176 vs paper #176
paper 0.6020 ±0.0110

Difference within reported uncertainty; not counted as underperformance.

paper 0.6020 ±0.0110 Δ at pub 0.0002
#ModelROC-AUC ↑Paper
1
GeomGCL
0.7630
2
LiGhT DeepGPT
0.7570 ±0.0110
3
CoMPT
0.7550
14
GCN
0.7350 ±0.0020
28
ContextPred*
actual #28 vs paper #171 143 places worse
0.7140 ±0.0180
best reported
actual #28 vs paper #171 143 places worse
paper 0.6260 ±0.0060
paper 0.6260 ±0.0060 Δ at pub 0.0880
31
JOAO*
actual #31 vs paper #167 136 places worse
0.7110 ±0.0140
best reported
actual #31 vs paper #167 136 places worse
paper 0.6280 ±0.0070
paper 0.6280 ±0.0070 Δ at pub 0.0830
34
GROVER
actual #34 vs paper #141 107 places worse
0.7080 ±0.0150
best reported
actual #34 vs paper #141 107 places worse
paper 0.6340 ±0.0060
paper 0.6340 ±0.0060 Δ at pub 0.0740
46
No Pretrain*
actual #46 vs paper #185 139 places worse
0.6913 ±0.0055
best reported
actual #46 vs paper #185 139 places worse
paper 0.6170 ±0.0050
paper 0.6170 ±0.0050 Δ at pub 0.0743
58
MoAMa
actual #58 vs paper #162 104 places worse
0.6801 ±0.0107
best reported
actual #58 vs paper #162 104 places worse
paper 0.6300 ±0.0040
64
GraphLoG*
actual #64 vs paper #137 73 places worse
0.6770 ±0.0190
best reported
actual #64 vs paper #137 73 places worse
paper 0.6350 ±0.0070
paper 0.6350 ±0.0070 Δ at pub 0.0420
114
Mole-BERT*
actual #114 vs paper #121 7 places worse
0.6450 ±0.0040
best reported
actual #114 vs paper #121 7 places worse
paper 0.6430 ±0.0020

Difference within reported uncertainty; not counted as underperformance.

paper 0.6430 ±0.0020 Δ today 0.0020
124
D-SLA*
actual #124 vs paper #125 1 place worse
0.6424 ±0.0050
best reported
actual #124 vs paper #125 1 place worse
paper 0.6420 ±0.0050

Difference within reported uncertainty; not counted as underperformance.

paper 0.6420 ±0.0050 Δ at pub 0.0004
129
MGSSL*
actual #129 vs paper #149 20 places worse
0.6410 ±0.0070
best reported
actual #129 vs paper #149 20 places worse
paper 0.6330 ±0.0050

Difference within reported uncertainty; not counted as underperformance.

paper 0.6330 ±0.0050 Δ at pub 0.0080
128
GraphMAE*
0.6410 ±0.0030
This paper
150
AttrMask*
actual #150 vs paper #150
0.6330 ±0.0060
actual #150 vs paper #150
paper 0.6330 ±0.0060

Improvement within reported uncertainty; not highlighted.

paper 0.6330 ±0.0060 Δ at pub 0.0040
170
Uni-Mol
0.6270 ±0.0040
This paper
194
MCM
0.6100 ±0.0040
This paper
#ModelRMSE ↓Paper
1
Graphormer-SPIS
0.4840 ±0.0050
2
Graphormer-SPD
0.4920 ±0.0040
3
conan-FGW
0.5140 ±0.0190
7
GCN
0.5640 ±0.0150
103
AttrMask
actual #103 vs paper #127 24 places worse
1.1120 ±0.0480
best reported
actual #103 vs paper #127 24 places worse
paper 1.4860 ±0.1200
paper 1.4860 ±0.1200 Δ at pub 0.3740
105
JOAO
actual #105 vs paper #136 31 places worse
1.1200
best reported
actual #105 vs paper #136 31 places worse
paper 1.6400 ±0.0900
paper 1.6400 ±0.0900 Δ at pub 0.5200 Δ today 0.5200
107
MoAMa
actual #107 vs paper #124 17 places worse
1.1250 ±0.0290
best reported
actual #107 vs paper #124 17 places worse
paper 1.3940 ±0.1200
113
MGSSL
actual #113 vs paper #126 13 places worse
1.1790 ±0.0080
best reported
actual #113 vs paper #126 13 places worse
paper 1.4380 ±0.1200
paper 1.4380 ±0.1200 Δ today 0.2590
114
ContextPred
actual #114 vs paper #135 21 places worse
1.1960 ±0.0370
best reported
actual #114 vs paper #135 21 places worse
paper 1.5970 ±0.1400
paper 1.5970 ±0.1400 Δ at pub 0.4010
130
GraphLoG
1.5070 ±0.1300
This paper
131
GraphMAE
1.5100 ±0.0900
This paper
134
D-SLA
1.5380 ±0.1300
This paper
135
MCM
1.5530 ±0.1100
This paper
136
No pretrain
1.6050 ±0.0800
This paper
137
Uni-Mol
1.7170 ±0.0800
This paper
139
Mole-BERT
1.7420 ±0.0500
This paper
152
GROVER
3.1070 ±1.3900
This paper
#ModelRMSE ↓Paper
1
SE(3)-I MPPH
0.3540 ±0.0530
2
ECFP-4 + RF
0.5600 ±0.0660
3
GPSE + GPS
0.6820
54
MoAMa
actual #54 vs paper #80 26 places worse
2.0720 ±0.0530
best reported
actual #54 vs paper #80 26 places worse
paper 2.7000 ±0.1100
67
GraphLoG
actual #67 vs paper #107 40 places worse
2.3350 ±0.0520
best reported
actual #67 vs paper #107 40 places worse
paper 3.4670 ±0.0600
69
GIN
2.3460 ±0.1220
83
GraphMAE
actual #83 vs paper #98 15 places worse
2.7400 ±0.2330
best reported
actual #83 vs paper #98 15 places worse
paper 3.0600 ±0.1500

Difference within reported uncertainty; not counted as underperformance.

paper 3.0600 ±0.1500 Δ today 0.3200
87
Uni-Mol
2.8110 ±0.1600
This paper
90
MGSSL
2.8400 ±0.1900
This paper
93
D-SLA
2.9200 ±0.0500
This paper
94
AttrMask
2.9340 ±0.1300
This paper
97
MCM
3.0510 ±0.1100
This paper
98
ContextPred
3.0570 ±0.1700
This paper
104
JOAO
actual #104 vs paper #107 3 places worse
3.2820 ±0.0020
best reported
actual #104 vs paper #107 3 places worse
paper 3.4670 ±0.0600
109
No pretrain
3.5890 ±0.3300
This paper
115
Mole-BERT
4.0290 ±0.0800
This paper
126
GROVER
7.9390 ±3.1700
This paper
#ModelRMSE ↓Paper
1
MemGNN
0.5560 ±0.0230
2
ProtoS-L2
0.5800 ±0.0160
3
ProtoW-L2
0.6040 ±0.0140
7
GCN
0.6550 ±0.0360
17
ContextPred
actual #17 vs paper #64 47 places worse
0.7020 ±0.0200
best reported
actual #17 vs paper #64 47 places worse
paper 0.8350 ±0.0100
paper 0.8350 ±0.0100 Δ at pub 0.1330
21
JOAO
actual #21 vs paper #91 70 places worse
0.7080
best reported
actual #21 vs paper #91 70 places worse
paper 1.0400 ±0.0500
paper 1.0400 ±0.0500 Δ at pub 0.3320 Δ today 0.3320
31
AttrMask
actual #31 vs paper #60 29 places worse
0.7300 ±0.0040
best reported
actual #31 vs paper #60 29 places worse
paper 0.8170 ±0.0100
paper 0.8170 ±0.0100 Δ at pub 0.0870
48
MoAMa
0.7720 ±0.0300
This paper
54
D-SLA
0.7900 ±0.0900
This paper
60
MCM
0.8150 ±0.1100
This paper
62
Mole-BERT
0.8210 ±0.0200
This paper
64
No pretrain
0.8310 ±0.0200
This paper
70
GraphMAE
0.8520 ±0.0200
This paper
73
MGSSL
0.8690 ±0.0600
This paper
82
GraphLoG
actual #82 vs paper #91 9 places worse
0.9320 ±0.0520
best reported
actual #82 vs paper #91 9 places worse
paper 1.0400 ±0.0500
96
Uni-Mol
1.1320 ±0.0500
This paper
103
GROVER
1.3770 ±0.0600
This paper
SOTA performance (not significantly different from the best) Proposed method (this paper) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green) Best today — OK at publication; today's leaderboard best is elsewhere