Large Language Model Meets Graph Neural Network in Knowledge Distillation

Recent advancements in leveraging Large Language Models (LLMs) for Text-Attributed Graphs (TAGs) learning have shown significant potential, but practical deployment is often hindered by substantial computational and storage demands. Conventional Graph Neural Networks (GNNs) are more efficient but struggle with the intricate semantics embedded in TAGs. To combine the semantic understanding of LLMs with the efficiency of GNNs, we propose a novel LLM-to-GNN knowledge distillation framework, istic raph nowledge istillation (LinguGKD), which employs TAG-oriented instruction tuning to train pre-trained LLMs as teachers and introduces a layer-adaptive contrastive distillation strategy to align node features between teacher LLMs and student GNNs within a latent space, effectively transferring the semantic and complex relational understanding from LLMs to GNNs. Extensive experiments across various LLM and GNN architectures on multiple datasets demonstrate that LinguGKD significantly enhances the predictive accuracy and convergence rate of GNNs without requiring additional training data or model parameters. Compared to teacher LLMs, the distilled GNNs offer superior inference speed and reduced resource requirements, making them highly practical for deployment in resource-constrained environments. Furthermore, our framework demonstrates significant potential for leveraging ongoing advancements in LLM research to continuously improve GNN performance.