LLM as Graph Kernel: Rethinking Message Passing on Text-Rich Graphs

Paper: 2603.14937 Authors: Ying Zhang, Hang Yu, Haipeng Zhang, Peng Di Categories: cs.LG, cs.CL

The Gap

Graph Neural Networks (GNNs) compress text into static embeddings before message passing. LLM-hybrid approaches use LLMs as feature extractors, then feed embeddings into GNN layers. Both create an information bottleneck: once text becomes a vector, you can’t go back to reason over the original words. Updates happen in embedding space, detached from the raw content that actually carries meaning.

The problem is architectural: we treat text as a node attribute to be encoded, not as the medium through which graph structure manifests. In citation networks, the text IS the relationship—one paper cites another because of specific claims in specific paragraphs. Compressing that into 768 dimensions throws away the very information message passing should propagate.

Problem: Text-rich graphs lose information in compression
   |
   v
Assumption: Text is the primary medium, not just an attribute
   |
   v
Method: LLM as aggregation operator, anchor on raw text
   |
   v
Evidence: Competitive performance without embedding bottleneck
   |
   v
Conclusion: LLMs can serve as graph kernels for structural reasoning

The Increment

One sentence: Before, LLMs extracted features then GNNs propagated; now, the LLM itself performs message passing directly on raw text.

Core Mechanism

RAMP maintains two representations per node: the original raw text (static anchor) and a dynamically updated message (compressed summary from neighbors). During each iteration, the LLM receives both: the node’s own text and aggregated messages from its neighbors. It generates a new message that gets sent to the next layer.

The key is that inference always happens on raw text. The LLM doesn’t reason over embeddings—it reads actual words. Messages are the only thing that flows through the graph structure, and they’re optimized to be compact (via learned compression) while preserving task-relevant information. The final prediction comes from the LLM processing the node’s raw text plus its final aggregated message.

For generative tasks, RAMP unifies everything under a generation framework. Instead of separate classification heads, it prompts the LLM to generate the answer directly. This means the same architecture handles both discriminative tasks (node classification) and generative tasks (question answering) without structural changes.

Iteration t:
                                                    
  Node i raw text ----+                            
                      |                            
                      v                            
                  [ LLM ]  <--- Messages from neighbors
                      |                            
                      v                            
                  New message -----> Propagate to neighbors
                      |                            
                      v                            
                  (repeat for t+1)                 
                                                    
Final prediction:                                  
  Node i raw text + Final aggregated message --> [LLM] --> Output

Think of it like a book club discussion. Each person (node) brings their own book (raw text) that never changes. During each round of discussion, people share short summaries (messages) of what they learned from their neighbors’ books. You always read your own book, but the summaries help you understand context. After several rounds, you write your final interpretation based on your book plus all the accumulated insights from the group. The summaries flow and evolve, but everyone’s original book stays intact—you can always go back to the source material.

Key Concepts

• Raw Text Anchoring: Instead of converting text to embeddings once and working in vector space, RAMP keeps the original text accessible throughout. Every time the LLM makes a decision, it reads the actual words. This is like keeping the original document open while taking notes, versus only working from your notes. If your notes miss something important, you can’t recover it from the vector—but with anchoring, the LLM can always reference the source. The cost is computational (LLM processes text repeatedly), but the benefit is no information loss from premature compression.

• LLM as Aggregation Operator: Traditional GNNs use mathematical operations (sum, mean, attention) to combine neighbor features. RAMP replaces this with an LLM call. The LLM receives messages from neighbors and generates a new message. This is fundamentally different from using LLMs as encoders. An encoder runs once per node; an aggregator runs once per node per iteration, and its output depends on graph structure. The LLM becomes the message passing mechanism itself, not a preprocessing step.

• Dual Representation Scheme: Each node has two things: immutable raw text (the anchor) and mutable messages (the propagated information). The raw text provides grounding—it’s the source of truth. Messages provide context—they’re compressed summaries of what neighbors know. During inference, both feed into the LLM. This separation lets you preserve full text fidelity while still enabling efficient propagation. Without it, you’d either lose text detail (pure GNN) or have no way to propagate information (pure LLM on isolated nodes).

Framework Shift

Before (GNN + LLM hybrid):        After (RAMP):

Text --> [LLM] --> Embedding      Text (static anchor)
            |                         |
            v                         v
      [GNN layers]              [LLM as aggregator]
       /    |    \                /    |    \
      v     v     v              v     v     v
   Propagate in              Messages propagate
   embedding space           (text stays raw)
            |                         |
            v                         v
      Prediction                Text + Messages --> [LLM] --> Output

[One sentence: From encoding text once then propagating vectors, to anchoring on raw text while propagating LLM-generated messages.]

Expert Assessment

Problem choice: Real gap. Text-rich graphs are everywhere (citation networks, knowledge graphs, social networks with posts), and the embedding bottleneck is a genuine limitation. The paper correctly identifies that existing LLM+GNN hybrids still compress too early. This sits at the intersection of two mature fields (GNNs and LLMs) and proposes a non-obvious integration.

Method maturity: Clever insight with practical constraints. The core idea—LLM as aggregation operator—is elegant. But the execution requires careful engineering: message compression to avoid context length explosion, prompt design for aggregation, handling computational cost. The paper doesn’t fully solve the efficiency problem (running LLMs iteratively is expensive), but it demonstrates the concept works. A simpler approach might be caching LLM representations with retrieval-augmented generation, but that reintroduces the bottleneck.

Experimental integrity: Baselines seem fair—they compare against both pure GNNs and recent LLM+GNN hybrids. The datasets (citation networks, product graphs) are appropriate for text-rich scenarios. One concern: the paper doesn’t deeply analyze when RAMP wins versus when embeddings suffice. If the text is simple or the task is purely structural, the overhead might not be worth it. The ablations show message passing helps, but more analysis of the cost-benefit tradeoff would strengthen the claims.

Writing quality: The abstract and introduction are strong—they clearly articulate the bottleneck problem. The method section gets dense with implementation details (prompt templates, message compression strategies) that could be streamlined. The related work section tries to cover too much ground. If they focused the related work on just the embedding bottleneck problem and moved other comparisons to appendix, the narrative would be tighter.

Verdict: weak accept — Solid idea with demonstrated feasibility, but needs more analysis of when the computational cost is justified and clearer guidelines for practitioners.

Takeaways

Dual representation pattern: Keep raw data accessible while propagating compressed summaries. This applies beyond graphs—any scenario where you need both full context and efficient propagation. For example, in multi-document QA, maintain original documents while passing around query-relevant snippets.

LLMs as operators, not just encoders: Stop thinking of LLMs as one-time feature extractors. They can be iterative components in larger systems. The key is designing what flows between iterations (here, messages) and what stays fixed (here, raw text).

Unified generative formulation: Instead of building separate heads for classification versus generation, frame everything as text generation with appropriate prompts. This simplifies architecture and potentially enables transfer between task types.

When to avoid this: If your text is short and simple, or if your task is primarily structural (like link prediction based on topology), the embedding bottleneck might not matter. RAMP’s overhead is only justified when text complexity and graph structure both matter significantly.