Motif-aware Attribute Masking for Molecular Graph Pre-training
Abstract
Attribute reconstruction is used to predict node or edge features in the pre-training of graph neural networks. Given a large number of molecules, they learn to capture structural knowledge, which is transferable for various downstream property prediction tasks and vital in chemistry, biomedicine, and material science. Previous strategies that randomly select nodes to do attribute masking leverage the information of local neighbors. However, the over-reliance of these neighbors inhibits the model's ability to learn long-range dependencies from higher-level substructures, such as functional groups or chemical motifs. To explicitly measure and encourage the inter-motif knowledge transfer in pre-trained models, we define inter-motif node influence measures and propose a novel motif-aware attribute masking strategy to capture long-range inter-motif structures by leveraging the information of atoms in neighboring motifs. Once each graph is decomposed into disjoint motifs, the features for every node within a sample motif are masked and subsequently predicted using a graph decoder. We evaluate our approach on eleven molecular classification and regression datasets and demonstrate its advantages.
Benchmark Results
Tested on 9 datasets 31 baseline values reported too low
Tables show presented results in global context (best 3 papers + GCN + MLP).
MoleculeNet
| # | Model | ROC-AUC ↑ | Paper |
|---|---|---|---|
| 1 | HIMP | 0.8740
±0.0050 | |
| 2 | GINE w/ VN | 0.8720
±0.0030 | |
| 3 | NaiveGINe+ K=3 w/ VN | 0.8700
±0.0040 | |
| 14 | GCN | 0.8400
±0.0040 | |
| 15 | GROVER | 0.8380
±0.0170 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 31 | JOAO | 0.8230
±0.0220 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 46 | ContextPred | 0.8060
±0.0120 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 72 | Uni-Mol | 0.7960
±0.0050 best reported | best today: A Systematic Survey and Benchmark of Deep Learning for Molecular Property Prediction in the Foundation Model Era (2026) Difference within reported uncertainty; not counted as underperformance. |
| 71 | GraphLoG | 0.7960
±0.0250 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 86 | No pretrain | 0.7920
±0.0199 best reported | best at pub: Does GNN Pretraining Help Molecular Representation? (NeurIPS 2022) |
| 112 | MoAMa | 0.7829
±0.0055 best reported | |
| 150 | Mole-BERT | 0.7710
±0.0040 best reported | best today: UniMAP: Universal SMILES-Graph Representation Learning (2023) Difference within reported uncertainty; not counted as underperformance. |
| 160 | D-SLA | 0.7681
±0.0052 best reported | best at pub: Graph Self-supervised Learning with Accurate Discrepancy Learning (NeurIPS 2022) Difference within reported uncertainty; not counted as underperformance. |
| 172 | MGSSL | 0.7650
±0.0030 best reported | best at pub: Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (NeurIPS 2021) |
| 202 | GraphMAE | 0.7550
±0.0060 | This paper |
| 229 | AttrMask | 0.7510
±0.0090 | prev best: A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining (ICML 2023) Improvement within reported uncertainty; not highlighted. |
| 251 | MCM | 0.7440
±0.0010 | This paper |
| # | Model | ROC-AUC ↑ | Paper |
|---|---|---|---|
| 1 | ProtoMol | 0.9140
±0.0030 | |
| 2 | MMSG | 0.9080
±0.0050 | |
| 3 | MemGNN | 0.9070 | |
| 22 | No pretrain | 0.8743
±0.0163 best reported | best at pub: Does GNN Pretraining Help Molecular Representation? (NeurIPS 2022) |
| 43 | GIN | 0.8646
±0.0081 | |
| 61 | Uni-Mol | 0.8570
±0.0020 best reported | best today: Invariant-Based Diagnostics for Graph Benchmarks (2026) |
| 94 | D-SLA | 0.8380
±0.0100 | This paper |
| 98 | GraphLoG | 0.8350
±0.0120 | prev best: Let Invariant Rationale Discovery Inspire Graph Contrastive Learning (ICML 2022) Improvement within reported uncertainty; not highlighted. |
| 100 | JOAO | 0.8340
±0.0330 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 105 | GraphMAE | 0.8310
±0.0090 | This paper |
| 118 | GROVER | 0.8260
±0.0070 best reported | best at pub: Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (NeurIPS 2021) |
| 130 | MGSSL | 0.8203
±0.0379 best reported | best at pub: Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (NeurIPS 2021) best today: Pre-training Graph Neural Networks on Molecules by Using Subgraph-Conditioned Graph Information Bottleneck (AAAI 2024) Difference within reported uncertainty; not counted as underperformance. |
| 143 | Mole-BERT | 0.8140
±0.0100 best reported | best today: UniMAP: Universal SMILES-Graph Representation Learning (2023) Difference within reported uncertainty; not counted as underperformance. |
| 147 | MoAMa | 0.8132
±0.0106 best reported | best today: Pre-training Graph Neural Networks on Molecules by Using Subgraph-Conditioned Graph Information Bottleneck (AAAI 2024) Difference within reported uncertainty; not counted as underperformance. |
| 182 | AttrMask | 0.7970
±0.0033 best reported | best at pub: A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining (ICML 2023) best today: A Generalist Cross-Domain Molecular Learning Framework for Structure-Based Drug Discovery (2025) Difference within reported uncertainty; not counted as underperformance. |
| 186 | ContextPred | 0.7960
±0.0120 best reported | best at pub: Graph Self-Contrast Representation Learning (2023) best today: Dual-perspective Cross Contrastive Learning in Graph Transformers (2024) Difference within reported uncertainty; not counted as underperformance. |
| 188 | MCM | 0.7950
±0.0130 | This paper |
| # | Model | ROC-AUC ↑ | Paper |
|---|---|---|---|
| 1 | MAGPrompt+ | 0.9432
±0.0238 | |
| 2 | CamS-LLaMA | 0.9420 | |
| 3 | SE(3)-I MPPH | 0.9400
±0.0210 | |
| 18 | No pretrain | 0.9223
±0.0307 best reported | best at pub: Does GNN Pretraining Help Molecular Representation? (NeurIPS 2022) |
| 34 | GIN | 0.8875
±0.0049 | |
| 37 | GROVER | 0.8870
±0.0060 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 39 | JOAO | 0.8790
±0.0200 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 40 | ContextPred | 0.8770
±0.0260 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 51 | MoAMa | 0.8589
±0.0061 best reported | |
| 54 | GraphLoG | 0.8460
±0.0080 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 61 | MGSSL | 0.7952
±0.0198 best reported | best at pub: Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (NeurIPS 2021) |
| 84 | Uni-Mol | 0.7290
±0.0060 best reported | best today: Invariant-Based Diagnostics for Graph Benchmarks (2026) |
| 89 | D-SLA | 0.7260
±0.0080 | This paper |
| 96 | Mole-BERT | 0.7230
±0.0070 best reported | best today: UniMAP: Universal SMILES-Graph Representation Learning (2023) Difference within reported uncertainty; not counted as underperformance. |
| 101 | GraphMAE | 0.7200
±0.0060 | This paper |
| 113 | MCM | 0.7160
±0.0060 | This paper |
| 141 | AttrMask | 0.7020
±0.0050 best reported | best at pub: 16 Pre-training Molecular Graph Representation with 3D Geometry (ICLR 2021) best today: Graph Generative Pre-trained Transformer (ICML 2025) |
| # | Model | ROC-AUC ↑ | Paper |
|---|---|---|---|
| 1 | SE(3)-I MPPH | 0.9930
±0.0040 | |
| 2 | KA-GCN | 0.9920
±0.0050 | |
| 3 | KA-GAT | 0.9910
±0.0050 | |
| 17 | Uni-Mol | 0.9190
±0.0180 best reported | |
| 19 | GCN | 0.9130
±0.0170 | |
| 28 | No pretrain | 0.8970
±0.0093 best reported | best at pub: Does GNN Pretraining Help Molecular Representation? (NeurIPS 2022) |
| 40 | AttrMask | 0.8770
±0.0119 best reported | |
| 56 | MoAMa | 0.8530
±0.0220 | This paper |
| 71 | GraphMAE | 0.8230
±0.0120 | This paper |
| 76 | GROVER | 0.8170
±0.0250 best reported | best at pub: Unified 2D and 3D Pre-Training of Molecular Representations (KDD 2022) best today: UniMAP: Universal SMILES-Graph Representation Learning (2023) Difference within reported uncertainty; not counted as underperformance. |
| 79 | JOAO | 0.8132
±0.0249 best reported | best at pub: Graph Self-Contrast Representation Learning (2023) |
| 85 | MGSSL | 0.8070
±0.0210 best reported | best at pub: Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (NeurIPS 2021) Difference within reported uncertainty; not counted as underperformance. |
| 87 | D-SLA | 0.8020
±0.0150 | prev best: Graph Self-supervised Learning with Accurate Discrepancy Learning (NeurIPS 2022) Improvement within reported uncertainty; not highlighted. |
| 88 | Mole-BERT | 0.8010
±0.0360 best reported | best today: UniMAP: Universal SMILES-Graph Representation Learning (2023) Difference within reported uncertainty; not counted as underperformance. |
| 103 | GraphLoG | 0.7872
±0.0258 best reported | best at pub: Let Invariant Rationale Discovery Inspire Graph Contrastive Learning (ICML 2022) best today: Graph Self-Contrast Representation Learning (2023) Difference within reported uncertainty; not counted as underperformance. |
| 115 | ContextPred | 0.7720
±0.0088 best reported | best at pub: A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining (ICML 2023) best today: A Generalist Cross-Domain Molecular Learning Framework for Structure-Based Drug Discovery (2025) Difference within reported uncertainty; not counted as underperformance. |
| 204 | MCM | 0.6470
±0.0050 | This paper |
| # | Model | ROC-AUC ↑ | Paper |
|---|---|---|---|
| 1 | KA-GAT | 0.8470
±0.0020 | |
| 2 | KA-GCN | 0.8420
±0.0010 | |
| 3 | GraphKAN | 0.8370
±0.0010 | |
| 37 | Uni-Mol | 0.6590
±0.0130 best reported | |
| 52 | GROVER | 0.6480
±0.0060 best reported | |
| 58 | MoAMa | 0.6460
±0.0050 | This paper |
| 66 | GIN | 0.6403
±0.0104 | |
| 68 | JOAO | 0.6400
±0.0100 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 90 | Mole-BERT | 0.6280
±0.0110 | This paper |
| 102 | MCM | 0.6230
±0.0090 | This paper |
| 101 | No pretrain | 0.6230
±0.0114 best reported | best at pub: Does GNN Pretraining Help Molecular Representation? (NeurIPS 2022) |
| 103 | ContextPred | 0.6220
±0.0059 best reported | best at pub: A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining (ICML 2023) best today: A Generalist Cross-Domain Molecular Learning Framework for Structure-Based Drug Discovery (2025) Difference within reported uncertainty; not counted as underperformance. |
| 116 | MGSSL | 0.6180
±0.0080 best reported | best at pub: Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (NeurIPS 2021) Difference within reported uncertainty; not counted as underperformance. |
| 123 | GraphLoG | 0.6150
±0.0130 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) Difference within reported uncertainty; not counted as underperformance. |
| 134 | AttrMask | 0.6120
±0.0012 best reported | best at pub: A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining (ICML 2023) best today: A Generalist Cross-Domain Molecular Learning Framework for Structure-Based Drug Discovery (2025) Difference within reported uncertainty; not counted as underperformance. |
| 175 | GraphMAE | 0.6030
±0.0110 | This paper |
| 176 | D-SLA | 0.6022
±0.0113 best reported | best at pub: Graph Self-supervised Learning with Accurate Discrepancy Learning (NeurIPS 2022) Difference within reported uncertainty; not counted as underperformance. |
| # | Model | ROC-AUC ↑ | Paper |
|---|---|---|---|
| 1 | GeomGCL | 0.7630 | |
| 2 | LiGhT DeepGPT | 0.7570
±0.0110 | |
| 3 | CoMPT | 0.7550 | |
| 14 | GCN | 0.7350
±0.0020 | |
| 28 | ContextPred* | 0.7140
±0.0180 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 31 | JOAO* | 0.7110
±0.0140 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 34 | GROVER | 0.7080
±0.0150 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 46 | No Pretrain* | 0.6913
±0.0055 best reported | best at pub: Does GNN Pretraining Help Molecular Representation? (NeurIPS 2022) |
| 58 | MoAMa | 0.6801
±0.0107 best reported | |
| 64 | GraphLoG* | 0.6770
±0.0190 best reported | best at pub: KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction (KDD 2022) |
| 114 | Mole-BERT* | 0.6450
±0.0040 best reported | best today: UniMAP: Universal SMILES-Graph Representation Learning (2023) Difference within reported uncertainty; not counted as underperformance. |
| 124 | D-SLA* | 0.6424
±0.0050 best reported | best at pub: Graph Self-supervised Learning with Accurate Discrepancy Learning (NeurIPS 2022) Difference within reported uncertainty; not counted as underperformance. |
| 129 | MGSSL* | 0.6410
±0.0070 best reported | best at pub: Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (NeurIPS 2021) Difference within reported uncertainty; not counted as underperformance. |
| 128 | GraphMAE* | 0.6410
±0.0030 | This paper |
| 150 | AttrMask* | 0.6330
±0.0060 | prev best: A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining (ICML 2023) Improvement within reported uncertainty; not highlighted. |
| 170 | Uni-Mol | 0.6270
±0.0040 | This paper |
| 194 | MCM | 0.6100
±0.0040 | This paper |
| # | Model | RMSE ↓ | Paper |
|---|---|---|---|
| 1 | Graphormer-SPIS | 0.4840
±0.0050 | |
| 2 | Graphormer-SPD | 0.4920
±0.0040 | |
| 3 | conan-FGW | 0.5140
±0.0190 | |
| 7 | GCN | 0.5640
±0.0150 | |
| 103 | AttrMask | 1.1120
±0.0480 best reported | best at pub: Improving Molecular Pretraining with Complementary Featurizations (2022) |
| 105 | JOAO | 1.1200 best reported | best at pub: 16 Pre-training Molecular Graph Representation with 3D Geometry (ICLR 2021) best today: 3D Molecular Geometry Analysis with 2D Graphs (2023) |
| 107 | MoAMa | 1.1250
±0.0290 best reported | |
| 113 | MGSSL | 1.1790
±0.0080 best reported | best today: Atom-Motif Contrastive Transformer for Molecular Property Prediction (2023) |
| 114 | ContextPred | 1.1960
±0.0370 best reported | best at pub: Improving Molecular Pretraining with Complementary Featurizations (2022) |
| 130 | GraphLoG | 1.5070
±0.1300 | This paper |
| 131 | GraphMAE | 1.5100
±0.0900 | This paper |
| 134 | D-SLA | 1.5380
±0.1300 | This paper |
| 135 | MCM | 1.5530
±0.1100 | This paper |
| 136 | No pretrain | 1.6050
±0.0800 | This paper |
| 137 | Uni-Mol | 1.7170
±0.0800 | This paper |
| 139 | Mole-BERT | 1.7420
±0.0500 | This paper |
| 152 | GROVER | 3.1070
±1.3900 | This paper |
| # | Model | RMSE ↓ | Paper |
|---|---|---|---|
| 1 | SE(3)-I MPPH | 0.3540
±0.0530 | |
| 2 | ECFP-4 + RF | 0.5600
±0.0660 | |
| 3 | GPSE + GPS | 0.6820 | |
| 54 | MoAMa | 2.0720
±0.0530 best reported | |
| 67 | GraphLoG | 2.3350
±0.0520 best reported | |
| 69 | GIN | 2.3460
±0.1220 | |
| 83 | GraphMAE | 2.7400
±0.2330 best reported | best today: Frequency-Corrupt Based Graph Self-Supervised Learning (WWW 2026) Difference within reported uncertainty; not counted as underperformance. |
| 87 | Uni-Mol | 2.8110
±0.1600 | This paper |
| 90 | MGSSL | 2.8400
±0.1900 | This paper |
| 93 | D-SLA | 2.9200
±0.0500 | This paper |
| 94 | AttrMask | 2.9340
±0.1300 | This paper |
| 97 | MCM | 3.0510
±0.1100 | This paper |
| 98 | ContextPred | 3.0570
±0.1700 | This paper |
| 104 | JOAO | 3.2820
±0.0020 best reported | |
| 109 | No pretrain | 3.5890
±0.3300 | This paper |
| 115 | Mole-BERT | 4.0290
±0.0800 | This paper |
| 126 | GROVER | 7.9390
±3.1700 | This paper |
| # | Model | RMSE ↓ | Paper |
|---|---|---|---|
| 1 | MemGNN | 0.5560
±0.0230 | |
| 2 | ProtoS-L2 | 0.5800
±0.0160 | |
| 3 | ProtoW-L2 | 0.6040
±0.0140 | |
| 7 | GCN | 0.6550
±0.0360 | |
| 17 | ContextPred | 0.7020
±0.0200 best reported | best at pub: Improving Molecular Pretraining with Complementary Featurizations (2022) |
| 21 | JOAO | 0.7080 best reported | best at pub: Improving Molecular Pretraining with Complementary Featurizations (2022) best today: 3D Molecular Geometry Analysis with 2D Graphs (2023) |
| 31 | AttrMask | 0.7300
±0.0040 best reported | best at pub: Improving Molecular Pretraining with Complementary Featurizations (2022) |
| 48 | MoAMa | 0.7720
±0.0300 | This paper |
| 54 | D-SLA | 0.7900
±0.0900 | This paper |
| 60 | MCM | 0.8150
±0.1100 | This paper |
| 62 | Mole-BERT | 0.8210
±0.0200 | This paper |
| 64 | No pretrain | 0.8310
±0.0200 | This paper |
| 70 | GraphMAE | 0.8520
±0.0200 | This paper |
| 73 | MGSSL | 0.8690
±0.0600 | This paper |
| 82 | GraphLoG | 0.9320
±0.0520 best reported | |
| 96 | Uni-Mol | 1.1320
±0.0500 | This paper |
| 103 | GROVER | 1.3770
±0.0600 | This paper |
