Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs

Paper: 2606.12385 Authors: Sanjay Adhikesaven, Haoxiang Sun, Sewon Min Categories: cs.CL

The Gap

Existing research on LLM provenance focuses on single-hop dependencies: “Model A was fine-tuned from Model B.” But modern LLM pipelines use models for data generation (e.g., GPT-4 outputting synthetic QA pairs), filtering (e.g., using a classifier to clean web text), judging (e.g., using GPT-4 as an evaluator of outputs), and guiding development decisions (e.g., using a reward model). These dependencies are multi-hop and recursive — Model A depends on Model B, which depended on Model C’s outputs during training, which in turn depended on Model D’s filtering decisions. The documentation of these chains is scattered across model cards, GitHub repos, paper appendices, and Hugging Face component registries. Prior work (e.g., tracing via model cards or manual inspection) stops at shallow depth because humans cannot feasibly follow 5+ hops across heterogeneous sources. The known unknown is the full structure of hidden dependencies. The paper fills this gap by formalizing dependency types and building an agentic system that recursively crawls and reconciles evidence across artifacts.

[Problem] --> [Assumption: dependencies are shallow and well-documented]
     v
[Reality: dependencies are deep, recursive, fragmented]
     v
[Prior approach: manual inspection of model cards]
     |
     v
[Limit: infeasible beyond 2 hops; misses indirect dependencies]
     |
     v
[This paper's method: formalize dependency types, agentic crawling with reconciliation]
     |
     v
[Evidence: 1060 source-verified dependencies from 4 large LLMs]
     |
     v
[Conclusion: hidden recursive dependencies, license chains, train-eval coupling exist at scale]

The Increment

One sentence: Before this paper, LLM dependency tracing was manual, shallow, and incomplete; after this paper, we have a formal framework and automated system to reconstruct multi-hop, recursively-verified dependency graphs with evidence citations.

Core Mechanism

ModSleuth is an agentic system. It begins with a seed artifact (e.g., a model release on Hugging Face). The system consists of three modules: Scanner, Reconciler, and Grapher. Scanner reads text from model cards, paper PDFs, GitHub READMEs, and configuration files to find references to other artifacts (e.g., “trained on data from Model X”). These references are raw strings: names, URLs, version tags. Reconciler takes these raw references and attempts to resolve them to canonical artifact identities by cross-referencing multiple sources — e.g., matching the string “LLaMA-2-7B” against Hugging Face IDs, GitHub releases, and paper citations. It also classifies the role of the dependency (training data, evaluation judge, filter, base model). Grapher builds a directed graph where nodes are artifacts and edges are typed dependency relationships. The system then recursively feeds each newly-identified artifact as a seed, performing depth-first search until no new references are found or the recursion depth exceeds a threshold.

[Seed Artifact] --> [Scanner] --(raw references)--> [Reconciler]
     ^                                                  |
     |                                              (canonical IDs,
     |                                               role labels)
     |                                                  v
     +------- [Grapher] <--- (edges with roles) --------+
                 |
                 v
        [New artifacts discovered]
                 |
                 v
        [Recursive scanning (depth limit)]

The structural metaphor is ancestry detective work. The Scanner is like a genealogist reading family letters, old photos, and census records to note every mention of “married to X” or “born in Y town.” The Reconciler is the archive cross-referencer: “Jane Doe” might appear in a notebook as “Aunt Jane” in a will, and “Jane Doe (née Smith)” in a church record — the reconciler figures out these all refer to the same person. The Grapher is the family tree builder, placing each person and linking them with relationship types (parent, spouse, cousin). Just as a genealogist then visits newly discovered relatives to check their records, ModSleuth recursively scans each new artifact, building a complete, evidence-backed dependency tree that no single person could compile manually.

Key Concepts

• Operation-centered dependency: Instead of saying “Model A depends on Model B,” the paper categorizes *how Model B is used: as a data generator (Model B outputs used for training), a filter (Model B decides what data to keep), a judge (Model B evaluates Model A’s outputs), or a base (Model A fine-tuned from Model B). This matters because responsibilities (like licenses) differ by role. Example: If Model A is released under a restrictive license, but its training data was generated by Model B which has a share-alike license, then Model A’s users might inherit obligations — an “operation-centered” view makes this visible. Prior work just said “A depends on B.”

• Artifact identity reconciliation: A single artifact (e.g., the CommonCrawl snapshot used in training GPT-2) might be called “cc100,” “commoncrawl 2019-02,” or simply “web crawl” in different documents. The Reconciler uses string normalization, fuzzy matching against known registries, and cross-document co-occurrence statistics to decide if two references point to the same entity. Without this, the dependency graph would be fractured into disconnected clusters of aliases.

• Recursive depth limit and termination condition: Because dependencies can be infinite (Model A -> B -> C -> …), ModSleuth sets a practical depth limit (e.g., 10 hops) and stops when no new artifacts are found for a given seed. This makes the problem tractable while still capturing long chains that human tracing would miss. The paper shows that even at depth 5, previously unconnected license obligations appear.

Framework Shift

Before (mainstream approach):        After (this paper):
[Manual, shallow tracing]            [Automated, recursive, evidence-backed]
+------------------+                  +--------------------+
| Model Card       |                  | Scanner (agent)    |
| "trained on X"   |                  | + Reconciler       |
| (1-2 hops)       |                  | + Grapher          |
+--------+---------+                  +---------+----------+
         |                                      |
         v                                      v
[Only explicit dependencies]         [All implicit, recursive, typed]
[No reconciliation of aliases]       [Alias resolution via cross-reference]
[No role classification]             [Operation-centered roles]

One sentence: From manual, shallow, role-agnostic tracing to automated, deep, role-sensitive reconstruction with evidence reconciliation.

Expert Assessment

Problem choice: Real gap. The paper addresses a growing infrastructure crisis in AI reproducibility and legal compliance. It sits at the intersection of systems and ethics, gaining urgency as regulatory scrutiny (e.g., EU AI Act) demands transparency. Not manufactured.

Method maturity: Clever engineering with formalization. The agentic crawling approach is not theoretically deep, but the formalization of dependency roles and artifact reconciliation is a genuine contribution that had been missing. A simpler brute-force approach (dumping all strings and doing regex matching) would miss the role classification and the reconciliation step — so the method is appropriately sophisticated.

Experimental integrity: The paper evaluates on four well-known LLM releases (including LLaMA, Mistral). The 1,060 dependencies are manually verified for a random subset — they report inter-annotator agreement but not the full breakdown. There is no comparison against a baseline automatic tracer (e.g., simply parsing model cards) because no such baseline existed. That is forgivable for a first system, but it means the reader cannot quantify the improvement over manual effort. The claim “discrepancies between released and training-time artifacts” is qualitative — missing a formal error rate.

Writing quality: Well-structured, but the introduction could be more concrete about the scale of the problem (give a mini case study of a real hidden chain). The “Related Work” section is thin; they should have contrasted with specific tools (e.g., Hugging Face model card extractor). The most disappointing part is the lack of a failure analysis — how often does the reconciler make a wrong match? Without that, the trustworthiness of the 1,060 dependencies is unclear.

Verdict: weak accept — The formalization of dependency roles and the agentic architecture are timely and useful, but the evaluation lacks rigorous ablation and error analysis. If a follow-up adds a quantitative benchmark, it becomes strong accept.

Takeaways

• Practitioners can adopt the “operation-centered dependency” framing for their own model cards: instead of saying “uses data from X,” specify whether X was a generator, filter, or judge. This small change would make downstream tracing far easier.
• The artifact reconciliation approach (string normalization + cross-registry fuzzy matching) is a transferable technique for any domain where entity resolution across heterogeneous metadata is needed (e.g., ML model registries, dataset versioning).
• The concept of “recursive dependency auditing” can be applied to other toolchains — for example, tracing which compiler versions and library dependencies exist in a deep learning framework’s build pipeline. The agentic scanning + reconciliation pattern generalizes.
• The paper’s release of dependency graphs is a concrete asset — teams can plug those graphs into license compliance tools (e.g., FOSSA for AI models) or dataset lineage trackers. No need to reinvent the mapping from scratch.