Tytan: Letting the Database Describe Itself

Paper: 2608.06331 Authors: Donna Hooshmand, Shubham Shahi, Cameron Barrie, Abhratanu Dutta, Marko Sterbentz, Harper Pack, Kristian J. Hammond Categories: cs.DB, cs.AI

The Gap

Here’s the thing nobody puts in the demo video. Every text-to-SQL system, every “ask your data a question” product, every automated dashboard generator sits on top of a layer that somebody wrote by hand. Call it the semantic layer, the metrics layer, the analytic schema — dbt has one, Looker has one (LookML), Cube has one, and every serious BI deployment has one. It says: customer is an entity, revenue is a measure not an identifier, order_id in this table joins to that table, and “monthly active users” means this specific aggregation over these specific rows.

That layer is the bottleneck. It’s written by the one person who understands both the business and the schema, it goes stale, and it’s wrong in ways nobody notices until a number on a slide is off by 40%.

What has prior work done? Two clusters, and they’re both partial:

Schema-side work — foreign-key discovery, inclusion-dependency mining, functional-dependency detection, entity matching. Decades of solid results here (think Sherlock, Sato, and the column-type-annotation line of work; the classic FK-discovery literature). It finds *structure but not meaning. It’ll tell you col_7 determines col_9; it won’t tell you col_7 is a customer and col_9 is their lifetime value, or that “customer” is even the right analytic unit.

LLM-side work — feed the DDL to a model, ask it to label columns, generate descriptions, or go straight to SQL. This gets you *meaning but with no guarantee the meaning connects to the data. The model happily asserts that revenue = price * quantity in a table where price is nullable 30% of the time, or generates a join path across a key that doesn’t actually satisfy inclusion. Text-to-SQL benchmarks (Spider, BIRD) mask this because the gold query exists — you never see the systematic failure mode of ungrounded semantic assertions.

The gap Tytan claims: nobody has built the thing that produces a complete, executable, semantically-typed analytic schema — and nobody has defined what it would even mean to evaluate one.

    PROBLEM
    [ analytic semantic layer is hand-written ]
    [ = knowledge acquisition bottleneck     ]
              |
              v
    WHY IT PERSISTED
    +-----------------------+     +------------------------+
    | symbolic methods:     |     | LLM methods:           |
    | find structure,       | AND | find meaning,          |
    | no meaning            |     | no grounding           |
    +-----------------------+     +------------------------+
              |                            |
              \____________  _____________/
                           \/
    ASSUMPTION
    [ the two are complementary, not competing ]
    [ symbolic = evidence,  LLM = hypothesis   ]
    [ residual ambiguity = ask the human       ]
              |
              v
    METHOD
    [ TYTAN: propose (LLM) -> verify (symbolic)      ]
    [        -> if still ambiguous, ask one question ]
              |
              v
    EVIDENCE  (three axes, deliberately separated)
    [ coverage         : 100% vs expert reference    ]
    [ retrieval exec   : 1678 / 1678 claims run     ]
    [ role accuracy    : 92 - 100%                  ]
    [ blind test       : 10-table live DB, no keys, ]
    [                    5 independent annotators   ]
              |
              v
    CONCLUSION
    [ the semantic layer can be machine-built, ]
    [ and "correct" is a measurable property   ]

The Increment

One sentence: Before this paper, “generate the semantic layer” was a prompt you wrote and eyeballed; after it, it’s a pipeline with a verification step and a three-axis definition of correctness that separates *did you find everything from does it execute from is it labeled right.

Core Mechanism

Tytan runs on a database plus, optionally, a short paragraph of user context (“this is our clinical trial enrollment system”). The pipeline has three functional stages, and the important architectural decision is who is allowed to assert what.

Symbolic analysis first. Before any LLM sees anything semantic, Tytan profiles the database: cardinalities, null rates, value distributions, uniqueness, and candidate inclusion dependencies. This matters enormously for the blind test — a live ten-table database with no declared foreign keys. Tytan has to discover that patient_ref in one table is contained in id of another, and it does this by checking the data, not by reading the DDL. The output is a pile of *evidence: facts about the data that are true by construction.

LLM proposal second. Now the model gets the profile plus column names plus sample values and proposes: what real-world entities live here (Patient, Enrollment, Site — not tbl_pt_enr_x), which columns are identifiers vs. measures vs. dimensions vs. temporal, which tables compose into a single unit of analysis, and what to call things in human language. Crucially, every proposal is a *claim, and every claim that touches data must come with a retrieval instruction — a concrete path from the schema element to the bytes.

Verification and the ask. Every retrieval instruction is executed. This is where the 1,678-of-1,678 number comes from: Tytan doesn’t publish a claim it hasn’t run. If a proposed join produces zero rows, or a “measure” turns out to be a categorical code, the claim dies. And when evidence genuinely underdetermines the answer — two plausible join paths, an ambiguous ID column that could be an entity key or a foreign reference — Tytan doesn’t guess and doesn’t silently pick. It asks the user one targeted natural-language question. This is the interactive part of “interactive neurosymbolic,” and it’s doing real work: it converts the system’s uncertainty into the cheapest possible human input rather than dumping a schema and hoping.

   INPUT
   +----------------------------+
   |  relational DB  (+ optional short description) |
   +----------------------------+
                |
                v
   .......... STAGE 1 : SYMBOLIC PROFILING ..........
   +------------------------------------------------+
   |  cardinality / uniqueness / null rate          |
   |  value distributions, sample rows              |
   |  candidate inclusion dependencies              |
   |     (works with NO declared FKs)               |
   +------------------------------------------------+
                |
                |  E V I D E N C E  (true by construction)
                v
   .......... STAGE 2 : LLM PROPOSAL ................
   +------------------------------------------------+
   |  entity proposal    : "this is a Patient"      |
   |  role assignment    : id / measure / dim / time|
   |  unit-of-analysis   : which tables compose     |
   |  naming             : human-readable labels    |
   +------------------------------------------------+
                |
                |  C L A I M S  (each carries a retrieval instruction)
                v
   .......... STAGE 3 : VERIFY ......................
   +------------------------------------------------+
   |   execute every retrieval instruction          |
   +------------------------------------------------+
        |              |                  |
      passes        fails             ambiguous
        |              |                  |
        |              v                  v
        |        [ discard /        [ ONE targeted
        |          re-propose ]       NL question
        |              |              to the user ]
        |              |                  |
        |              +--------+---------+
        |                       |
        |                       v  (answer becomes evidence,
        |                       |   re-enter stage 2)
        v                       |
   +----------------------------+-------------------+
   |   ANALYTIC SEMANTIC SCHEMA                     |
   |   entities + roles + names + verified paths    |
   +------------------------------------------------+
                |
                v
   consumed by: NL query interfaces, auto-reports, BI

The metaphor: an archaeologist, a philologist, and a site supervisor.

You’ve excavated a site. Thousands of potsherds, a few walls, no labels.

The archaeologist is the symbolic stage. She doesn’t interpret; she measures. This sherd is 4cm thick. These two fragments have edges that physically fit. This wall course sits below that one, so it’s older. Everything she writes down is verifiable by anyone who picks up the object. She will never tell you the building was a temple — that’s not her job. In Tytan, this is the profiler: patient_ref values are 100% contained in patients.id, amount has 40,000 distinct values and no nulls.

The philologist is the LLM. He reads the inscriptions, knows a hundred other sites, and says: “given the layout and this word here, this was a granary, that room was an office, and this pit is a votive deposit.” He’s producing *interpretation, and he’s often right because he’s seen the pattern before. But he is capable of confident nonsense — reading a shopping list as a hymn. In Tytan: tbl_pt_enr_x is Enrollment, amount is a measure you can sum, Patient and Enrollment compose into one analytic unit.

The site supervisor is the verification loop, and she’s the reason the report is trustworthy. Every claim the philologist makes, she checks against the archaeologist’s measurements. “You say these two rooms connect — go walk from one to the other.” That’s executing the retrieval instruction. If the doorway is bricked up, the claim is struck. And when the evidence genuinely doesn’t decide — this could be a granary or a stable, the artifacts fit both — she doesn’t flip a coin and she doesn’t write “granary (probably)”. She walks over to the one local farmer who’s been on this land for sixty years and asks him one specific question. That’s the targeted NL question, and note how cheap it is: not “please review this 200-page report,” but “was there ever livestock in this corner?”

The load-bearing part of the analogy: the final site report contains only claims that survived the supervisor. That’s why 1,678 of 1,678 execute — not because the philologist is infallible, but because nothing gets into the report unchecked.

Key Concepts

• Analytic semantic schema (vs. the schema you already have): Your database schema says orders.customer_id INTEGER REFERENCES customers(id). That’s a *storage description. An analytic schema says something different: “the thing analysts care about is a Customer, a Customer has a lifetime spend which you compute by summing orders.total grouped by customer_id, and orders.status is a category you can filter on but never average.” Same data, different vocabulary. The gap between them is exactly why a text-to-SQL model can produce syntactically valid SQL that answers the wrong question. The storage schema knows the plumbing; the analytic schema knows the *questions.

• Retrieval instruction as the unit of verifiability: This is the paper’s sharpest move, and it transfers well beyond databases. Instead of asking “is this label correct?” (needs a human, subjective), the system requires every semantic claim to carry an *executable consequence. If you claim total_spend is an aggregable feature of Customer, you must emit the query that computes it. Then correctness has a cheap mechanical proxy: does it run and return sane rows? This turns a large chunk of a fuzzy annotation problem into a compiler problem. The analogy is doctests — a comment claiming a function returns a sorted list is unverifiable prose; a doctest showing sort([3,1]) == [1,3] either passes or doesn’t.

• Ambiguity as a first-class output, not an error: Most pipelines have exactly two modes: succeed, or fail. Tytan has three: verified, rejected, and *underdetermined-so-ask. That third state is the interesting one. It requires the system to know the difference between “I don’t know” and “the data cannot know” — an ID column with no declared reference and two plausible parent tables is genuinely ambiguous no matter how good your model is. No amount of scale fixes it; the information isn’t in the database. Recognizing that and routing it to one sentence of human input is more honest than a confidence score.

Framework Shift

  Before (mainstream approach):          After (this paper):

  == PATH A: symbolic ==                 +---------------------------+
                                         |  DATABASE  (the world)    |
  [ DB ] --> [ FK discovery ]            +---------------------------+
             [ dep. mining  ]                    |
                  |                              v
                  v                        [ MEASURE ]  symbolic
            structure, no meaning          evidence, true by
            ( col_7 -> col_9 )             construction
                                                 |
  == PATH B: LLM ==                              v
                                           [ INTERPRET ] LLM
  [ DDL ] --> [ prompt ] --> [ labels ]     hypotheses, each
                              [ SQL   ]     bundled with an
                  |                         executable claim
                  v                              |
            meaning, no grounding                v
            ( plausible, unchecked )       [ EXECUTE ]
                                            pass / fail / ???
  == and the human ==                            |
                                          +------+------+
  [ expert ] --> writes 500 lines               |      |
                 of LookML by hand           ship   ask ONE
                 (weeks, stale, wrong)               question
                                                     |
  no shared definition of                            v
  "is this schema correct?"                     [ SCHEMA where
                                                  every claim
                                                  has been run ]

  eval: vibes, or downstream            eval: coverage / execution /
        text-to-SQL accuracy                  role accuracy, split

One sentence: From *generate a schema and trust it to generate hypotheses and let the database adjudicate, the core shift is treating LLM output as claims requiring evidence rather than answers requiring formatting — and treating leftover ambiguity as a question to ask rather than a guess to make.

Expert Assessment

Problem choice: Real gap, and well-timed. This is genuinely the unglamorous blocker in the entire natural-language-analytics stack. Everyone benchmarks text-to-SQL on Spider and BIRD, where the schema is small and the semantics are self-evident from column names. Point those same systems at a real enterprise warehouse with 400 tables named dw_fct_txn_agg_v2 and they collapse — not because SQL generation is hard, but because *nobody told the model what the business means. Attacking the semantic layer instead of the query generator is the right call, and it’s where the field is heading. The authors are also from a group (Hammond’s) with a long history in this exact problem space, so the framing has real operational grounding rather than benchmark-chasing.

Method maturity: Clever architecture, not clever algorithms. There’s no new inference procedure here — the components are standard: data profiling, inclusion-dependency checking, prompted LLM proposals, execution as validation. The insight is the *contract: no semantic claim ships without an executable consequence. That’s a genuinely good idea, cheap to implement, and it’s the reason the numbers look the way they do. But be clear-eyed: an experienced engineer with a weekend and a decent prompt could build 70% of this. The remaining 30% — the ambiguity detection, the question generation, the composition of tables into analytic units — is where the real work is, and unfortunately that’s the part the abstract describes least.

Is something simpler being overlooked? Possibly. The paper doesn’t say how much the symbolic stage actually contributes on databases that do have declared foreign keys. On Spider-style schemas, a well-prompted LLM reading the DDL might match Tytan’s entity structure. The blind test (no declared keys) is the case where symbolic profiling clearly earns its keep, and it’s also the only test where that’s demonstrated. An ablation would settle this; I don’t see one advertised.

Experimental integrity: Mixed, and this is where I’d push hardest.

The good: the three-axis decomposition (coverage / retrieval correctness / characterization accuracy) is a real methodological contribution. Separating “did you find it,” “does it run,” and “is it labeled right” prevents the usual mush where one metric hides three failure modes. The blind test with five independent annotators is the strongest single piece of evidence in the paper — held out, live, no declared keys, multiple judges. That’s the design of someone trying to be caught out rather than trying to win.

The concerning: 100% coverage across seven domains is a number that should make you suspicious, not impressed. Perfect scores usually mean the metric is measuring something easier than it sounds. Two questions the abstract doesn’t answer. First, the references are “expert-corrected” — corrected by whom, and with what visibility into Tytan’s output? If the reference was built by an expert reviewing Tytan’s proposals, coverage is close to circular. Second, coverage against a reference measures recall only. What about the entities Tytan proposes that *aren’t in the reference? A system that hallucinates fifteen spurious entities and one real one scores 100% coverage. I want precision, and I want the false-positive rate. Its absence from the abstract is a choice.

Then there’s this line: “Checking the underlying data showed the small disagreement is in the reference, not in TYTAN.” I want to believe it, and it might be entirely true — but “we lost 8% on this metric, then re-examined and concluded the ground truth was wrong” is structurally the most dangerous sentence in empirical work. It should be adjudicated by someone blind to which system produced which answer. If it wasn’t, that 92–100% range is really “92% measured, argued up to 100%.”

And 1,678-of-1,678 retrieval instructions executing is a weaker claim than it reads. These are self-generated claims, verified by the same execution loop that filters them. Of course they all execute — that’s the pipeline’s invariant, not an experimental finding. The interesting numbers would be: how many claims were proposed and discarded before these 1,678 survived? What’s the retry rate? And does “executes correctly” mean “returns rows” or “returns the right rows”? A join that silently produces a Cartesian product executes beautifully.

Finally, no baselines are named in the abstract. Not GPT-4-with-a-good-prompt, not existing commercial semantic-layer generators, not the FK-discovery literature. For a systems paper this is somewhat forgivable — the contribution is partly “here’s a working artifact and a way to evaluate it” — but without a comparison, the reader can’t tell how much of the performance comes from the architecture and how much from a strong underlying model.

Writing quality: The abstract is doing a lot of work and doing it well — the three-axis framing is stated crisply and the blind test is properly foregrounded. But the corner-cutting is visible in exactly the place it matters. The ambiguity-detection mechanism is the intellectual core (it’s in the title, twice over: *interactive and neurosymbolic), and it gets one sentence: “asks the user a targeted natural-language question.” When? On what criterion? How many questions per database — three, or thirty? Because if it’s thirty, this is an expert-assisted tool with good ergonomics, not an automation story, and that changes the entire value proposition. Rewriting that section with a decision rule, a question budget, and a sensitivity analysis on answer quality would elevate the whole paper from “nice system” to “reusable principle.” Second priority: an honest precision/false-positive table alongside the coverage numbers.

Verdict: weak accept — the problem is real, the propose-verify-ask architecture is a sound and reusable design, and the blind test is well constructed; but the near-perfect headline numbers are under-interrogated, precision is missing, and the ambiguity mechanism that makes the title’s claim is the least specified part of the paper.

Takeaways

Things worth stealing, whether or not you touch databases:

Make every semantic claim carry an executable consequence. This is the transferable core. Anywhere an LLM produces structured output — API specs, config, ontologies, data dictionaries, extraction schemas — require it to emit not just the assertion but a runnable check for that assertion, then run it before shipping. You convert “is this label right?” (needs a human) into “does this execute?” (needs a CPU). You won’t catch semantic errors, but you’ll catch the entire class of *ungrounded errors, which in practice is most of them.

Split your metric before you’re forced to. Coverage, executability, and label accuracy fail for different reasons and have different fixes. A single “schema quality” score would have hidden all three. When you’re evaluating a generative system on a fuzzy task, spend an afternoon decomposing what “correct” means — the decomposition is often more durable than the system.

Build the three-state pipeline. Verified / rejected / underdetermined-so-ask. Most LLM systems only have two states and paper over the third with confidence scores, which don’t distinguish “the model is unsure” from “the information isn’t present.” If your input genuinely underdetermines the output, one well-targeted question is worth more than any amount of model capacity. Design for the question, and design it to be answerable in one sentence.

Profile before you prompt. Give the model computed facts about the data (cardinality, null rates, distinct values, actual sample rows) rather than just the DDL. Cheap to compute, and it moves the model from guessing-from-names to reasoning-from-evidence. The blind test here — a live database with no declared foreign keys — is the realistic case, and it’s the case where names alone lie to you.

If you’re building NL-to-data: the lesson is that your bottleneck probably isn’t the SQL generator. Spider and BIRD trained a generation of researchers to optimize query synthesis on schemas where semantics were free. In production, semantics are the expensive part. Fix the semantic layer first.

One caution to carry forward: when your system scores 100%, your first move should be to attack your own metric, not write it in the abstract. Ask what the perfect score is *not measuring. Here, it’s precision — and that omission is the thing a reviewer will find first.