CHARM: Building Graph Foundation Models That Actually Transfer Across Multimodal Domains

Paper: 2607.26023 Authors: Ankang Yang, Jitao Zhao, Di Jin, Yuxiao Huang, Dongxiao He Categories: cs.AI

The Gap

Graph foundation models have been on everyone’s radar lately — the idea of pretraining one big model that transfers across graph tasks and domains is compelling. But there’s a messy reality that prior work has been sidestepping. GNN-based foundation models (think of anything building on Graphormer, GPS, or similar architectures) learn general graph inductive biases, but when you drop them into a new domain — say, a new social network or a new product graph — they still need fine-tuning. They can’t do zero-shot transfer. On the flip side, recent LLM-based graph methods (GraphGPT, LLaGA, etc.) are better at zero-shot generalization, but they mostly handle unimodal graphs — nodes have text or IDs, not both text AND images AND structure simultaneously. They also tend to be trained and tested within a single domain, so the “transfer” is really just generalization to new tasks in familiar territory.

The gap is precise: nobody has built a graph foundation model that (a) natively handles multimodal node features (text + images + graph topology), and (b) can do genuine zero-shot transfer to unseen graph domains without any fine-tuning. This paper targets that exact intersection.

Real-world graphs have                 Existing GFMs need
multimodal node features       -->     downstream adaptation
(text + images + structure)            (GNN-based: Graphormer, GPS)
        |                                      |
        v                                      v
Multimodal graphs need         <--     LLM-based methods handle
zero-shot cross-domain transfer         only unimodal / single-domain
        |                                      |
        +------ TWO CORE CHALLENGES -----------+
        |                                      |
        v                                      v
Challenge 1: Transfer knowledge          Challenge 2: Disentangle
across modalities and capture            domain-specific patterns
cross-modal relations                    from shared concepts
        |                                      |
        +----------- CHARM --------------------+
        |                                      |
        v                                      v
Hierarchical graph contexts          Modality-aware encoder +
replace isolated nodes               LLM for concept abstraction
        |                                      |
        +---------- EVIDENCE ------------------+
        |                                      |
        v                                      v
Consistent improvements on zero-shot multimodal graph tasks

The Increment

One sentence: Before CHARM, you either had graph models that generalized across tasks but needed fine-tuning on new domains, or LLM-based models that did zero-shot but couldn’t handle multimodal graphs across domains — CHARM closes both gaps simultaneously.

Core Mechanism

CHARM’s architecture has three main stages. First, instead of treating each node as a raw feature vector, it constructs hierarchical graph contexts — think of these as nested neighborhoods that capture information at multiple scales. A node doesn’t just know about its immediate neighbors; it knows about its local cluster, its community, and how those communities relate to each other. These contexts encode both the structural topology and the multimodal features (text descriptions, images) associated with nodes.

Second, a modality-aware graph context encoder takes these hierarchical contexts and does something clever: it processes each modality separately, then fuses them with graph structure information. The output is a set of graph tokens — dense vector representations that summarize the rich multimodal context around each node. These tokens are designed to be domain-agnostic; they map the messy, domain-specific patterns (this particular social network’s structure, that particular product graph’s image features) into a shared concept space.

Third, these graph tokens are fed into a large language model as a prefix or context. The LLM already knows about concepts from its pretraining; the graph tokens give it structured, multimodal grounding. The LLM then performs downstream reasoning — classification, link prediction, etc. — without any gradient updates to its parameters. Zero-shot, no fine-tuning.

Multimodal Input Graph
========================
Nodes with text, images, edges
        |
        v
+-----------------------------+
| Hierarchical Context Builder |
|                             |
| Layer 1: Local neighbors    |
| Layer 2: Community-level    |
| Layer 3: Cross-community    |
+-----------------------------+
        |
        v
+-----------------------------+
| Modality-Aware Encoder      |
|                             |
| Text branch  --+            |
| Image branch --+--> Fusion  |---> Graph Tokens
| Graph branch --+            |
+-----------------------------+
        |
        v
+-----------------------------+
| Large Language Model        |
| (frozen parameters)         |
|                             |
| Graph Tokens + Task Prompt  |
|        |                    |
|        v                    |
| Zero-shot prediction        |
+-----------------------------+

Here’s a structural metaphor that makes this click: imagine you’re a diplomat arriving in a new country (unseen domain) where you don’t speak the language and don’t know the customs. Old approaches would force you to attend a local school (fine-tuning) before you can function. CHARM is like giving you a universal briefing book. The hierarchical context builder is like your intelligence team: they don’t just tell you about the capital city (immediate neighbors), they also map the provinces (communities) and how trade routes connect them (cross-community relations). The modality-aware encoder is like your team of specialist advisors — one reads the local newspapers (text), one studies satellite imagery (images), one analyzes infrastructure maps (graph structure) — and they merge their reports into a single briefing that highlights universal concepts: “this is a market town,” “this is a border region,” etc. The LLM is you, the experienced diplomat: you already know how markets and border regions work from other countries (pretraining). You apply that general knowledge to this briefing and make decisions without ever attending local school. The key insight is that the briefing book (graph tokens) is designed to speak in concepts you already understand, not in this country’s specific dialect.

Key Concepts

• Hierarchical Graph Context: Think of looking at a city on Google Maps. Zoom in: you see individual buildings (local context — immediate neighbors). Zoom out: you see neighborhoods (community context). Zoom out more: you see how neighborhoods connect via highways (cross-community context). Traditional GNNs mostly operate at one zoom level — they aggregate from immediate neighbors. CHARM explicitly builds contexts at multiple zoom levels and concatenates them. Why does this matter for transfer? Because high-level concepts like “hub node” or “bridge between communities” exist in almost every graph domain, regardless of what the nodes represent. By capturing these multi-scale patterns, the model learns representations that are less tied to any specific domain’s details.

• Modality-Aware Encoding: Imagine you’re at a party and someone hands you a person’s business card (text), shows you their photo (image), and tells you who they came with (graph structure). You process each piece of information differently — you read the card, you glance at the photo, you mentally map the social connections. Then you form an integrated impression. CHARM’s encoder does exactly this: separate processing branches for each modality, then a fusion mechanism that respects how modalities relate to each other. The “modality-aware” part means the fusion isn’t just concatenation — it uses the graph structure to guide how modalities are combined (e.g., if two nodes are connected, their image features should be compared, not just independently encoded).

• Zero-Shot Graph Transfer: In NLP, zero-shot means you train on task A and test on task B without any examples from B. In graphs, zero-shot transfer is harder because graph domains vary wildly — a social network looks nothing like a citation network or a molecular graph. The paper’s contribution is showing that by abstracting domain-specific node patterns into high-level concepts via hierarchical contexts, the model can recognize patterns like “this node is central” or “these two nodes share a role” even in graphs it has never seen. The LLM acts as the final reasoning engine that bridges the gap between these abstract graph concepts and natural-language task descriptions.

Framework Shift

Before (mainstream approach):        After (this paper):
                                     +=========================+
Raw node features                    | Hierarchical Graph      |
(vectors or text only)               | Contexts                |
        |                            | (multi-scale,           |
        v                            |  multimodal)            |
+------------------+                 +=========================+
| Single-modality  |                              |
| GNN encoder      |                              v
+------------------+                 +-------------------------+
        |                            | Modality-Aware Encoder  |
        v                            | (text+image+structure   |
+------------------+                 |  fused via graph)       |
| Domain-specific  |                 +-------------------------+
| adaptation /     |                              |
| fine-tuning      |                              v
+------------------+                 +-------------------------+
        |                            | Frozen LLM              |
        v                            | (graph tokens as input) |
   Task output                       +-------------------------+
                                              |
                                              v
                                    Zero-shot task output
                                    (no adaptation needed)

From domain-specific single-modality models that require fine-tuning to a multimodal hierarchical context abstraction that plugs into a frozen LLM for zero-shot transfer — the core shift is treating nodes not as isolated feature vectors but as rich, multi-scale, cross-modal neighborhoods that can be mapped to universal concepts.

Expert Assessment

Problem choice: This is a genuine and well-motivated gap. The intersection of multimodal graphs and zero-shot transfer has been conspicuously underexplored. Most graph foundation model work either stays in unimodal land or accepts the need for fine-tuning. The paper correctly identifies that real-world graphs are inherently multimodal (a product has text reviews, images, and purchase-graph structure) and that labeling every new domain is impractical. This sits right at the frontier of where graph ML needs to go.

Method maturity: The hierarchical context construction is the clever part — it’s a principled way to extract domain-invariant features. The modality-aware encoder is more standard (separate encoders + fusion), but the integration with graph structure to guide fusion is a nice touch. The LLM-as-reasoning-engine component is borrowed from recent work and isn’t novel per se, but the way graph tokens are constructed to be LLM-compatible is well-designed. One concern: the computational cost of building hierarchical contexts for large graphs isn’t fully addressed. There might be simpler approximations being overlooked.

Experimental integrity: The paper shows consistent improvements across zero-shot multimodal graph tasks, which is promising. However, I’d want to see: (a) ablation studies isolating the contribution of hierarchical context vs. modality-aware encoding vs. LLM backbone, (b) comparison against simply concatenating multimodal features with a strong GNN and using CLIP-style alignment, and (c) scalability experiments on graphs with millions of nodes. The baselines appear reasonable but the zero-shot multimodal graph benchmark space is nascent — there’s a risk of favorable benchmark selection.

Writing quality: The paper is clearly structured and the motivation flows well. The weakest section is likely the experimental analysis — the authors could strengthen the paper significantly by including failure case analysis (when does CHARM’s abstraction lose important domain-specific information?) and by providing clearer guidance on hyperparameter sensitivity across different graph types.

Verdict: weak accept — The problem is real and important, the approach is principled rather than brute-force, and the zero-shot multimodal graph setting is novel enough to warrant attention. But the experimental evidence needs more depth and the scalability story remains incomplete.

Takeaways

Three concrete things to steal:

1. Hierarchical context construction for domain-invariant features: The idea of replacing raw node features with multi-scale neighborhood summaries before any downstream processing is broadly useful. If you’re working on any transfer learning problem with graph data (not just multimodal), building hierarchical contexts as a preprocessing step can help disentangle domain-specific from domain-general patterns.

2. Modality-aware fusion guided by graph structure: Instead of naively concatenating features from different modalities, use the graph topology to determine which modalities should interact and how. If two nodes are connected, their cross-modal features should be compared; if they’re distant, keep them independent. This principle applies to any multimodal graph task, not just foundation models.

3. Graph tokens as LLM-compatible abstractions: The paper demonstrates a practical blueprint for bridging graph data and LLMs — abstract graph neighborhoods into fixed-size tokens that encode structural and multimodal semantics in a way LLMs can reason over. This template is reusable for anyone trying to build LLM-powered graph applications.