Graph-Bert: Only Attention is Needed for Learning Graph Representations

The dominant graph neural networks (GNNs) over-rely on the graph links, several serious performance problems with which have been witnessed already, e.g., suspended animation problem and over-smoothing problem. What's more, the inherently inter-connected nature precludes parallelization within the graph, which becomes critical for large-sized graph, as memory constraints limit batching across the nodes. In this paper, we will introduce a new graph neural network, namely Graph-Bert (Graph based Bert), solely based on the attention mechanism without any graph convolution or aggregation operators. Instead of feeding Graph-Bert with the complete large input graph, we propose to train Graph-Bert with sampled linkless subgraphs within their local contexts. Graph-Bert can be learned effectively in a standalone mode. Meanwhile, a pre-trained Graph-Bert can also be transferred to other application tasks directly or with necessary fine-tuning if any supervised label information or certain application oriented objective is available. We have tested the effectiveness of Graph-Bert on several graph benchmark datasets. Based the pre-trained Graph-Bert with the node attribute reconstruction and structure recovery tasks, we further fine-tune Graph-Bert on node classification and graph clustering tasks specifically. The experimental results have demonstrated that Graph-Bert can out-perform the existing GNNs in both the learning effectiveness and efficiency.