Multi-hop Attention Graph Neural Networks

Self-attention mechanism in graph neural networks (GNNs) led to state-of-the-art performance on many graph representation learning tasks. Currently, at every layer, a node computes attention independently for each of its graph neighbors. However, such attention mechanism is limited as it does not consider nodes that are not connected by an edge but can provide important network context information. Here we propose Multi-hop Attention Graph Neural Network (MAGNA), a principled way to incorporate multi-hop context information into every layer of GNN attention computation. MAGNA diffuses the attention scores across the network, which increases the ``receptive field'' for every layer of the GNN. Unlike previous approaches, MAGNA uses a diffusion prior on attention values, to efficiently account for all paths between the pair of not-connected nodes. We demonstrate theoretically and experimentally that MAGNA captures large-scale structural information in every layer, and has a low-pass effect that eliminates noisy high-frequency information from the graph. Experimental results on node classification as well as knowledge graph completion benchmarks show that MAGNA achieves state-of-the-art results: MAGNA achieves up to $5.7\