TopoBench: A Framework for Benchmarking Topological Deep Learning

L. Telyatnikov, Guillermo Bernardez, Marco Montagna, Mustafa Hajij, Martin Carrasco, Pavlo Vasylenko, Mathilde Papillon, Ghada Zamzmi, Michael T. Schaub, Jonas Verhellen, Pavel Snopov, Bertran Miquel-Oliver, Manel Gil-Sorribes, Alexis Molina, V. Guallar, Theodore Long, Julian Suk, Patryk Rygiel, Alexander Nikitin, Giordan Escalona, M. Banf, Dominik Filipiak, Max Schattauer, Liliya Imasheva, Álvaro Martínez, Halley Fritze, Marissa Masden, Valentina S'anchez, M. Lecha, Andrea Cavallo, C. Battiloro, Matt Piekenbrock, M. Tec, George Dasoulas, N. Miolane, Simone Scardapane, Theodore Papamarkou

Main figure for TopoBench: A Framework for Benchmarking Topological Deep Learning

Abstract

This work introduces, an open-source library designed to standardize benchmarking and accelerate research in topological deep learning (TDL). decomposes TDL into a sequence of independent modules for data generation, loading, transforming and processing, as well as model training, optimization and evaluation. This modular organization provides flexibility for modifications and facilitates the adaptation and optimization of various TDL pipelines. A key feature of is its support for transformations and lifting across topological domains. Mapping the topology and features of a graph to higher-order topological domains, such as simplicial and cell complexes, enables richer data representations and more fine-grained analyses. The applicability of is demonstrated by benchmarking several TDL architectures across diverse tasks and datasets.

Benchmark Results

Tested on 5 datasets 4 baseline values reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

Heterophily Benchmark

#ModelROC-AUC ↑Paper
1
BuNN
0.9899 ±0.0016
2
DC-GNN
0.9850 ±0.0021
3
Tango -GPS
0.9839 ±0.0054
36
GAT
actual #36 vs paper #93 57 places worse
0.9391 ±0.0035
best reported
actual #36 vs paper #93 57 places worse
paper 0.8964 ±0.0043
paper 0.8964 ±0.0043 Δ at pub 0.0237 Δ today 0.0427
42
GraphSAGE
0.9358 ±0.0005
68
GCN
actual #68 vs paper #106 38 places worse
0.9150 ±0.0006
best reported
actual #68 vs paper #106 38 places worse
paper 0.8752 ±0.0042
paper 0.8752 ±0.0042 Δ at pub 0.0223 Δ today 0.0398
82
SCN
0.9032 ±0.0011
This paper
95
CCCN
0.8942
This paper
98
SCCNN
0.8900
This paper
101
CWN
0.8862 ±0.0004
This paper
104
GIN
0.8782 ±0.0034
This paper
112
EDGNN
0.8452 ±0.0005
This paper
117
AST
0.8114 ±0.0005
This paper
121
UniGNN2
0.7802
This paper
156
MLP
0.5099 ±0.0147
#ModelROC-AUC ↑Paper
1
Polynormer-r
0.8591 ±0.0074
2
DC-GNN
0.8588 ±0.0081
3
TANGO-GPS
0.8566 ±0.0101
28
GAT
actual #28 vs paper #28
0.8443 ±0.0100
actual #28 vs paper #28
paper 0.8443 ±0.0100

Improvement within reported uncertainty; not highlighted.

paper 0.8443 ±0.0100 Δ at pub 0.0073
50
GCN
actual #50 vs paper #62 12 places worse
0.8364 ±0.0067
best reported
actual #50 vs paper #62 12 places worse
paper 0.8302 ±0.0071

Difference within reported uncertainty; not counted as underperformance.

paper 0.8302 ±0.0071 Δ at pub 0.0062 Δ today 0.0062
60
AST
0.8326 ±0.0010
This paper
76
SAGE
0.8243 ±0.0044
89
GIN
actual #89 vs paper #89
0.8072 ±0.0119
actual #89 vs paper #89
paper 0.8072 ±0.0119
paper 0.8072 ±0.0119 Δ at pub 0.0667
124
EDGNN
0.7753 ±0.0001
This paper
125
UniGNN2
0.7735 ±0.0003
This paper
139
MLP
0.7362 ±0.0020

Other Graph Benchmarks

#ModelAccuracy ↑Paper
1
MA-GCL
0.8883 ±0.0030
2
CCA-SSG
0.8842 ±0.0030
3
ARIEL
0.8827 ±0.0020
9
GAT
actual #9 vs paper #32 23 places worse
0.8693 ±0.0030
best reported
actual #9 vs paper #32 23 places worse
paper 0.5017 ±0.0059
paper 0.5017 ±0.0059 Δ at pub 0.3676
10
GCN
actual #10 vs paper #32 22 places worse
0.8651 ±0.0050
best reported
actual #10 vs paper #32 22 places worse
paper 0.4956 ±0.0055
paper 0.4956 ±0.0055 Δ at pub 0.3695
30
CWN
0.5190 ±0.0015
This paper
31
AST
0.5050 ±0.0027
This paper
32
CCCN
0.5026 ±0.0017
This paper
33
GIN
0.4916 ±0.0102
This paper
34
UniGNN2
0.4906 ±0.0008
This paper
35
EDGNN
0.4818 ±0.0009
This paper
38
SAGE
0.4648

Hypergraph Benchmarks

#ModelAccuracy ↑Paper
1
HAN (full batch)
0.8404 ±0.0102
2
AllSetTransformer
actual #2 vs paper #6 4 places worse
0.8363 ±0.0147
best reported
actual #2 vs paper #6 4 places worse
paper 0.8219 ±0.0261
best at pub: Hypergraph Benchmarks (ICLR 2021)

Difference within reported uncertainty; not counted as underperformance.

paper 0.8219 ±0.0261 Δ at pub 0.0144
3
UniGCNII
0.8360 ±0.0114
6
EDGNN
0.8236 ±0.0072
This paper
9
UniGNN2
0.8171 ±0.0142
This paper
20
GCN
0.7516 ±0.0065
21
MLP
0.7431 ±0.0189
#ModelAccuracy ↑Paper
2
AllSetTransformer
0.9198 ±0.0018
This paper
1
HealHGNN
0.9198 ±0.0031
3
ED-HNN
0.9190 ±0.0019
14
EDGNN
0.9083 ±0.0025
This paper
15
UniGNN2
0.9072 ±0.0023
This paper
20
MLP
0.8483 ±0.0022
SOTA performance (not significantly different from the best) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green) Best at pub — reported value improved over the best known when published