ICLR 2021

Hypergraph Benchmarks

Eli Chien, Chao Pan, Jianhao Peng, O. Milenkovic

Main figure for Hypergraph Benchmarks

Abstract

Hypergraphs are used to model higher-order interactions amongst agents and there exist many practically relevant instances of hypergraph datasets. To enable the efficient processing of hypergraph data, several hypergraph neural network platforms have been proposed for learning hypergraph properties and structure, with a special focus on node classification tasks. However, almost all existing methods use heuristic propagation rules and offer suboptimal performance on benchmarking datasets. We propose AllSet, a new hypergraph neural network paradigm that represents a highly general framework for (hyper)graph neural networks and for the first time implements hypergraph neural network layers as compositions of two multiset functions that can be efficiently learned for each task and each dataset. The proposed AllSet framework also for the first time integrates Deep Sets and Set Transformers with hypergraph neural networks for the purpose of learning multiset functions and therefore allows for significant modeling flexibility and high expressive power. To evaluate the performance of AllSet, we conduct the most extensive experiments to date involving ten known benchmarking datasets and three newly curated datasets that represent significant challenges for hypergraph node classification. The results demonstrate that our method has the unique ability to either match or outperform all other hypergraph neural networks across the tested datasets: As an example, the performance improvements over existing methods and a new method based on heterogeneous graph neural networks are close to $4\

Proposed method: MPNN

Benchmark Results

Tested on 4 datasets

Tables show presented results in global context (best 3 papers + GCN + MLP).

Hypergraph Benchmarks

#ModelAccuracy ↑Paper
1
HealHGNN
0.7718 ±0.0235
2
Deep-HGNN
0.7526 ±0.0176
3
mechanism (shared)
0.7470 ±0.0756
8
HAN (full batch)
0.7105 ±0.0226
This paper
10
AllSetTransformer
actual #10 vs paper #10
0.6983 ±0.0522
best reported
actual #10 vs paper #10
paper 0.6933 ±0.0220

Difference within reported uncertainty; not counted as underperformance.

paper 0.6933 ±0.0220 Δ today 0.0050
11
CEGAT
0.6909 ±0.0300
This paper
12
MLP
0.6793 ±0.0233
This paper
13
AllDeepSets
0.6782 ±0.0240
This paper
14
HNHN
0.6780 ±0.0259
This paper
15
UniGCNII
0.6725 ±0.0257
This paper
16
CECGN
0.6280 ±0.0261
This paper
17
HGNN
0.6139 ±0.0296
This paper
18
HCHA
0.6136 ±0.0253
This paper
19
HyperGCN
actual #19 vs paper #22 3 places worse
0.6013 ±0.0176
best reported
actual #19 vs paper #22 3 places worse
paper 0.4831 ±0.0293
paper 0.4831 ±0.0293 Δ today 0.1182
#ModelAccuracy ↑Paper
1
HAN (full batch)
0.8404 ±0.0102
This paper
2
AllSetTransformer
0.8363 ±0.0147
This paper
3
UniGCNII
0.8360 ±0.0114
This paper
4
HGNN
0.8264 ±0.0165
This paper
5
HCHA
0.8255 ±0.0097
This paper
8
AllDeepSets
0.8197 ±0.0150
This paper
10
HyperGCN
0.7948 ±0.0208
This paper
12
HNHN
0.7719 ±0.0149
This paper
13
CECGN
0.7705 ±0.0126
This paper
15
CEGAT
0.7616 ±0.0119
This paper
20
GCN
0.7516 ±0.0065
21
MLP
0.7431 ±0.0189
This paper
#ModelAccuracy ↑Paper
1
HealHGNN
0.9198 ±0.0031
2
AllSetTransformer
0.9198 ±0.0018
3
ED-HNN
0.9190 ±0.0019
6
UniGCNII
0.9169 ±0.0019
This paper
8
MPNN
0.9153 ±0.0023
This paper
9
AllDeepSets
0.9127 ±0.0027
This paper
11
HGNN
0.9103 ±0.0020
This paper
12
HCHA
0.9092 ±0.0022
This paper
13
HAN (full batch)
0.9089 ±0.0023
This paper
16
HyperGCN
actual #16 vs paper #16
0.8998 ±0.0043
best reported
actual #16 vs paper #16
paper 0.8938 ±0.0025

Difference within reported uncertainty; not counted as underperformance.

paper 0.8938 ±0.0025 Δ today 0.0060
17
CEGAT
0.8859 ±0.0029
This paper
18
CECGN
0.8800 ±0.0026
This paper
19
HNHN
0.8678 ±0.0029
This paper
20
MLP
0.8483 ±0.0022
This paper
#ModelAccuracy ↑Paper
1
HealHGNN
0.6822 ±0.0056
2
ED-HNN
0.6679 ±0.0041
3
AllSetTransformer
0.6546 ±0.0025
This paper
5
AllDeepSets
0.6455 ±0.0033
This paper
7
HCHA
0.6245 ±0.0026
This paper
8
HGNN
0.6200 ±0.0024
This paper
10
UniGCNII
0.5445 ±0.0037
This paper
11
CECGN
0.5444 ±0.0024
This paper
12
HNHN
actual #12 vs paper #15 3 places worse
0.5118 ±0.0035
best reported
actual #12 vs paper #15 3 places worse
paper 0.4718 ±0.0035
paper 0.4718 ±0.0035 Δ today 0.0400
13
CEGAT
0.5114 ±0.0056
This paper
14
HyperGCN
actual #14 vs paper #16 2 places worse
0.4874 ±0.0281
best reported
actual #14 vs paper #16 2 places worse
paper 0.4474 ±0.0281

Difference within reported uncertainty; not counted as underperformance.

paper 0.4474 ±0.0281 Δ today 0.0400
15
HAN (mini batch)
0.4857 ±0.0104
This paper
16
MLP
0.4551 ±0.0024
This paper
SOTA performance (not significantly different from the best) Proposed method (this paper) Baseline / comparison (this paper) Global context (other papers) Best today — OK at publication; today's leaderboard best is elsewhere