SkillCenter: Building a Verified Knowledge Base for AI Agents at Scale

Paper: 2607.07676 Authors: Tianming Sha, Yue Zhao, Lichao Sun, Yushun Dong Categories: cs.AI

The Gap

Here’s the situation: autonomous AI agents have gotten remarkably good at doing things — calling APIs, writing code, orchestrating multi-step workflows. Frameworks like ReAct, Toolformer, and Voyager gave agents the ability to reason and act. But there’s a dirty secret: these agents often operate on unverified operational knowledge. They can generate plausible-looking code or configuration that turns out to be wrong, insecure, or unmaintainable. The knowledge is there — scattered across ArXiv papers, Stack Overflow threads, GitHub repos, vendor docs — but it’s not structured, not quality-filtered, and definitely not traceable.

Prior skill libraries tried to solve pieces of this. Voyager built domain-specific skill sets for Minecraft. DEPS focused on hierarchical planning in constrained environments. GITM tackled game-specific knowledge. But all of them were narrow in domain, small in scale, and offered zero guarantees about where their knowledge came from. Community knowledge bases (GitHub, forums) are vast but noisy. Academic knowledge is rigorous but locked in papers. Nobody had built a bridge between these worlds at scale with quality guarantees.

SkillCenter fills exactly this gap: it’s the first large-scale, open skill library that combines source-grounded academic knowledge with curated community contributions, and every claim is formally traceable to an exact quotation in its source.

Problem                    Assumption                 Method                     Evidence                   Conclusion
|                          |                          |                          |                          |
v                          v                          v                          v                          v
Agents execute             LLMs can reliably          Multi-source               216,938 skills             Source-grounded
tasks but operate          judge quality of           pipeline: acquire          across 24 domains;         skills provide
on unverified,             domain knowledge           + SkillGate filter         SkillGate achieves         traceable, higher-
untraceable knowledge      claims                     + template + source        high precision on          quality operational
|                          |                          grounding                  quality filtering          knowledge for agents
v                          v                          |                          v                          |
Prior skill libs           Source-grounding           v                          Offline-searchable         v
were small, narrow,        as a formal guarantee      SQLite FTS5 bundles        community + academic       Verdict: real gap,
not source-traced          enables trust              for practical use          integration                practical contribution

The Increment

One sentence: Before this paper, agents had access to either small curated skill sets or large unverified knowledge; after this paper, they have a 216K-skill library where every factual claim is formally traceable to its source quotation.

Core Mechanism

SkillCenter’s pipeline has five stages that transform raw, messy source material into clean, searchable, source-grounded skill bundles. First, multi-source acquisition pulls content from peer-reviewed journals, ArXiv, and over 24,000 technical sources (yielding 114,565 academic skills), alongside community contributions from GitHub and the ClawHub marketplace (102,373 skills). These two streams represent fundamentally different trust profiles: academic sources come pre-validated but need structuring; community sources are rich but noisy.

The critical innovation is SkillGate, an LLM-based quality filter that sits between raw acquisition and final publishing. SkillGate evaluates each candidate skill against multiple criteria — factual accuracy, actionability, security implications, maintainability — and rejects or flags skills that don’t meet the bar. After filtering, template-driven generation standardizes skills into a uniform structured format so they’re interoperable. Iterative source-grounding then ensures every retained claim in each skill maps to an exact quotation in its original source document. Finally, quality-controlled publishing bundles everything into domain-specific SQLite FTS5 databases that support offline full-text search.

[Raw Sources]                    [Pipeline]                         [Output]
|                                |                                  |
+-- ArXiv/Journals ------------->|                                  |
|   (114,565 academic)           v                                  |
|                        +----------------+                         |
+-- GitHub/ClawHub ----->| Multi-source   |                         |
    (102,373 community)  | Acquisition    |                         |
                         +-------+--------+                         |
                                 |                                  |
                                 v                                  |
                         +----------------+                         |
                         | SkillGate      |--- reject ---> [trash]  |
                         | (LLM quality   |                         |
                         |  filter)       |                         |
                         +-------+--------+                         |
                                 |                                  |
                                 v                                  |
                         +----------------+                         |
                         | Template       |                         |
                         | Standardization|                         |
                         +-------+--------+                         |
                                 |                                  |
                                 v                                  |
                         +----------------+                         |
                         | Source         |                         |
                         | Grounding      |                         |
                         | (quote each    |                         |
                         |  claim)        |                         |
                         +-------+--------+                         |
                                 |                                  |
                                 v                                  |
                         +----------------+    +-----------+         |
                         | Quality-       |--->| SQLite    |         |
                         | Controlled     |    | FTS5      |         |
                         | Publishing     |    | Bundles   |         |
                         +----------------+    | (24       |         |
                                               | domains)  |         |
                                               +-----------+         |

Think of this like a university library system that’s been tasked with building a reference collection for a new AI assistant. Multi-source acquisition is like the acquisitions department ordering textbooks from academic publishers and also accepting donated materials from the community. SkillGate is the head librarian who actually reads every book before it hits the shelves — checking whether the claims are supportable, whether the advice is safe, whether the information is current. Template standardization is the cataloging department applying Dewey Decimal or Library of Congress classification so everything has a uniform call number and can be found on the same system. Source grounding is the footnoting policy: every factual claim in the catalog entry must have a page number pointing to the exact passage in the book. SQLite FTS5 bundles are the card catalogs — each domain gets its own physical cabinet so you don’t have to search the entire library when you just need the networking section. The whole point: when the AI agent pulls a skill off the shelf, it’s not just a plausible-sounding recipe. It’s a verified, cited, searchable piece of operational knowledge with a receipt.

Key Concepts

• Source Grounding: Imagine you’re a journalist. Every fact in your article needs a source. Source grounding in SkillCenter works the same way: if a skill says “use --no-new-privileges flag for Docker security,” that claim must be backed by an exact quotation from a source document — not paraphrased, not inferred, but directly quoted. This creates a verifiable chain from the skill’s advice to its origin. The practical value: when a source becomes outdated (say a library version changes), you can trace which skills are affected. Without this, you have a pile of unverifiable assertions.

• SkillGate: Think of this as a bouncer at a very selective club, except the bouncer has read everything and evaluates each applicant on multiple dimensions. An LLM evaluates candidate skills for actionability (can an agent actually use this?), correctness (is this advice sound?), security (does this introduce vulnerabilities?), and maintainability (will this age well?). The key insight is that LLMs can serve as automated quality judges when given clear criteria — not perfect judges, but good enough to filter the worst noise from community contributions at scale.

• Skill Bundles as SQLite FTS5: This is a deliberately pragmatic design choice. Instead of building a custom vector database or requiring a running server, SkillCenter ships skills as SQLite files with full-text search. Each domain gets its own file. An agent can search locally, offline, without any API calls. This sounds boring, but it’s actually brilliant for deployment: it works in air-gapped environments, it has zero infrastructure cost, and SQLite is the most widely deployed database engine on Earth.

Framework Shift

Before (mainstream approach):          After (this paper):

  Agent                                Agent
    |                                    |
    v                                    v
+-----------+                      +-----------+
| Reasoning |                      | Reasoning |
| Engine    |                      | Engine    |
+-----------+                      +-----------+
    |                                    |
    v                                    v
+-----------+                      +-----------+
| Tools /   |                      | SkillCenter|
| APIs      |                      | (verified, |  <-- NEW LAYER
+-----------+                      |  sourced   |
    |                              |  knowledge)|
    v                              +-----------+
+-----------+                          |
| Implicit  |                          v
| knowledge |                      +-----------+
| (unverified,                     | Tools /   |
|  untraced)                       | APIs      |
+-----------+                      +-----------+
                                       |
                                       v
                                   +-----------+
                                   | Implicit  |
                                   | knowledge |
                                   | (backstop)|
                                   +-----------+

From relying on an agent’s implicit, unverifiable knowledge to equipping agents with a curated, source-traceable knowledge layer that sits between reasoning and execution.

Expert Assessment

Problem choice: This is a real gap. The agent ecosystem has been so focused on reasoning and tool-use that the knowledge quality problem has been largely ignored. As agents move from demos to production, the “where did this advice come from?” question becomes critical — especially in security-sensitive domains. The problem is well-positioned in the field’s trajectory: we’ve built the engines, now we need fuel that doesn’t explode.

Method maturity: More pragmatic than clever. The LLM-as-quality-gate (SkillGate) is a reasonable approach but introduces a dependency on the gate-keeper model’s own reliability — who watches the watchmen? There’s no adversarial evaluation of SkillGate’s failure modes. The source grounding mechanism is the genuine contribution: it’s a simple idea (quote your sources) but hard to execute at scale, and they seem to have done it. I suspect simpler approaches (e.g., just filtering by citation count or venue quality) were overlooked in favor of the more sophisticated LLM pipeline.

Experimental integrity: The abstract and available information emphasize scale (216K skills, 24 domains) as the headline metric. Scale is not quality. The paper would benefit enormously from: (a) human evaluation of SkillGate’s precision/recall on a held-out set, (b) downstream task performance showing agents using SkillCenter actually produce better outputs, and (c) comparison against naive baselines like “just give the agent the raw source documents.” Without these, the 216K number is impressive but not dispositive.

Writing quality: The abstract is dense but well-structured. My main concern: the framing leans heavily on scale as a proxy for value. The sentence “to our knowledge the largest open skill library for agents by total count” is doing a lot of work — but count alone doesn’t tell us if the library is useful. Section 3 (the pipeline) would benefit from a frank discussion of failure cases: what does SkillGate reject incorrectly? How many false positives slip through?

Verdict: weak accept — The source-grounding formalization is a genuine and transferable contribution, and the scale is real. But the paper needs stronger empirical evidence that agents actually perform better with SkillCenter than without it, and that SkillGate’s quality filtering works better than simpler heuristics.

Takeaways

Three things worth stealing:

1. Source grounding as a design pattern: The idea that every claim in a knowledge base should be traceable to an exact quotation is powerful and domain-general. If you’re building any kind of agent knowledge system — for healthcare, legal, DevOps — adopt this pattern. It turns your knowledge base from a black box into an auditable system.

2. SQLite FTS5 as an agent knowledge store: The choice to ship knowledge as offline-searchable SQLite files is a lesson in deployment pragmatism. If you’re building agent tools, stop over-engineering the retrieval layer. SQLite works everywhere, costs nothing, and the full-text search is good enough for structured skill lookup.

3. The two-stream integration model: Combining high-trust academic sources with high-volume community sources, then filtering both through a quality gate, is a pattern that transfers to any domain where you have “official” and “community” knowledge. Think: internal docs + Slack knowledge in enterprise, or medical guidelines + clinical forum discussions in healthcare AI.