teLLMe Why: Turning Dashcam Data into Causal Questions with LLMs

Paper: 2607.15254 Authors: Qiwei Li, Jorge Ortiz Categories: cs.AI, cs.HC

The Gap

Traffic agencies are drowning in data but starving for causal insight. Dashcams, traffic cameras, and fleet telematics produce massive observational datasets — and everyone wants to answer questions like “Does rain actually increase congestion, or is it just correlated with rush hour?” The problem: the tools that could answer these questions (DoWhy, causal structure learning, DAG-based adjustment) require PhD-level statistical fluency. Traffic engineers and urban planners aren’t causal inference specialists. Meanwhile, simpler approaches — correlation analysis, basic regression — can’t distinguish causation from confounding, and running randomized experiments on real roads is obviously impractical.

The prior landscape splits into two camps that don’t talk to each other well: (1) causal inference tools that are powerful but require users to manually specify DAGs, choose adjustment sets, and validate assumptions — a steep learning curve, and (2) LLM-based data analysis tools that can parse natural language questions but lack causal reasoning capabilities, happily reporting spurious correlations as findings. This paper sits at the intersection: can we build a system where domain experts ask causal questions in plain English, and the system does the heavy lifting of structure learning, stability checking, and effect estimation — while being honest about what it can and can’t claim?

                         Observational traffic data
                                    |
                                    v
          +-------------------------------------------+
          | Existing approaches: two broken paths     |
          |                                           |
          | Path A: Simple stats    Path B: Causal    |
          | (regression,            inference tools    |
          |  correlation)           (DoWhy, PC algo)  |
          |         |                     |           |
          |         v                     v           |
          |  Confusing           Requires expert      |
          |  correlation         to specify DAGs,     |
          |  for causation       adjust sets,         |
          |                      assumptions          |
          +-------------------------------------------+
                                    |
                                    v
                    Gap: no accessible bridge
                    from NL question to
                    causal answer for
                    domain experts
                                    |
                                    v
                    teLLMe: LLM parses question,
                    PC algorithm learns structure,
                    bootstrap validates stability,
                    DoWhy estimates effects
                                    |
                                    v
                    "Causal Card" with estimates,
                    assumptions, and uncertainty
                    made explicit
                                    |
                                    v
                    Hypothesis generation,
                    not definitive claims

The Increment

One sentence: Before this paper, causal analysis of urban driving data required either deep statistical expertise or acceptance of spurious correlations; after this paper, domain experts can ask natural-language causal questions and receive structured, assumption-transparent causal reports.

Core Mechanism

teLLMe has five main stages that form a pipeline from raw video-derived data to natural-language causal answers. The first stage is data structuring: dashcam video annotations (from BDD100K-style datasets) are converted into a structured event table where each row is an event (a traffic jam, a weather condition, a time-of-day label, traffic density measures). This table becomes the observational dataset for all downstream analysis.

The second stage is query translation. A schema-aware LLM takes a natural-language question like “How does rain affect traffic density during peak hours?” and maps it to a formal causal query: specifying the treatment variable (rain), the outcome (traffic density), and any subpopulation constraints (peak hours). The “schema-aware” part is crucial — the LLM knows what columns exist in the event table, so it can ground its translation in actual data fields rather than hallucinating variables.

The third stage is causal structure learning using the PC algorithm. The PC algorithm starts with a fully connected graph of all variables and systematically tests conditional independence relationships — if two variables become independent after conditioning on a third, the edge between them is removed. This produces a partially directed acyclic graph (PDAG) representing plausible causal relationships. The fourth stage adds bootstrap stability checks: the structure learning is repeated on many resampled versions of the data, and only edges that appear consistently across resamples are treated as reliable. This prevents the system from reporting fragile, noise-driven relationships.

The fifth stage is query-specific effect estimation. Given the learned causal structure and the specific treatment/outcome from the translated query, the system identifies appropriate adjustment sets (covariates that must be controlled for to estimate the causal effect) and uses linear regression or DoWhy’s estimation methods to compute the effect size. Everything is packaged into a Causal Card — a structured report containing the effect estimate, the adjustment set used, the DAG support (which edges were found), the assumptions being made, and a short natural-language explanation generated by the LLM.

 Dashcam Annotations
        |
        v
 +------------------+
 | Event Table      |   structured rows: event_id,
 | (structured)     |   weather, time, density, ...
 +------------------+
        |
        v
 +------------------+
 | Schema-Aware LLM |   NL question ->
 | Query Translation|   (treatment, outcome, subpop)
 +------------------+
        |
        v
 +------------------+
 | PC Algorithm     |   conditional independence tests
 | (structure learn)|   -> partially directed DAG
 +------------------+
        |
        v
 +------------------+
 | Bootstrap        |   resample N times,
 | Stability Check  |   keep consistent edges
 +------------------+
        |
        v
 +------------------+
 | Effect           |   identify adjustment set,
 | Estimation       |   regress or DoWhy estimate
 +------------------+
        |
        v
 +------------------+
 | Causal Card      |   effect size, assumptions,
 | + NL explanation  |   DAG, uncertainty
 +------------------+

The structural metaphor: Imagine you’re a detective investigating whether a suspect (rain) caused a crime (traffic congestion). You have hundreds of hours of surveillance footage (dashcam data), but nobody filmed an experiment — you only have what happened naturally on the streets.

First, your crime scene technicians (the event table pipeline) go through the footage and create a standardized report for each incident: time, weather, traffic volume, location, outcome. Now you have a filing cabinet of comparable cases.

Then you bring in a smart assistant (the schema-aware LLM) who speaks your language. You ask: “Did rain cause the congestion?” The assistant translates this into precise investigative terms: we’re looking at rain as the suspect, congestion as the outcome, and we should check if it happened during rush hour (a potential confounder).

Next comes the forensic analyst (the PC algorithm). Instead of assuming the suspect is guilty, the analyst looks at *all the evidence systematically. If traffic congestion always happens at 5pm regardless of rain, then rain isn’t the real cause — the analyst removes that false link. If congestion only happens when it rains AND it’s rush hour, that’s a different story. The analyst builds a map of which variables influence which — a causal diagram.

But one analysis might be wrong. So the replication team (bootstrap stability) runs the same forensic analysis on 1000 slightly different samplings of the evidence. If a link only appears in some samples, it’s probably noise. Only links that survive repeated testing get reported.

Finally, the prosecutor (effect estimation) asks the specific question: given what we now know about the causal structure, how much *does rain increase congestion? Using the right statistical controls (adjustment sets), they produce a number — with a clear list of what assumptions they’re making.

The verdict (the Causal Card) doesn’t say “rain caused this traffic jam.” It says: “Based on our analysis, rain is associated with X% increase in congestion after controlling for time of day. Here’s our causal diagram, here’s what we assumed, and here’s our uncertainty. Use this to guide your investigation, not as the final word.”

Key Concepts

• Conditional Independence: This is the bedrock idea. Two variables are conditionally independent given a third if, once you know the third variable, knowing one of the others tells you nothing new about the other one. Concrete example: shoe size and reading ability are correlated in children (bigger shoes → better reading). But they’re conditionally independent given age — among 8-year-olds, shoe size tells you nothing about reading skill. The PC algorithm uses this logic in reverse: if conditioning on a variable makes an edge disappear, that edge was probably a confounding artifact, not a real causal link.

• Confounding and Adjustment Sets: A confounder is a variable that causes both the treatment and the outcome, creating a spurious association. Classic example: ice cream sales and drowning deaths are correlated, but neither causes the other — hot weather (the confounder) causes both. To estimate the true causal effect of one variable on another, you must “adjust for” (statistically control for) confounders. An adjustment set is the specific set of variables you control for. TeLLMe’s structure learning automatically identifies which variables need adjustment, rather than requiring the user to guess.

• Causal Card: This is the paper’s framing device for output. Rather than just returning a number (“rain increases congestion by 23%”), the Causal Card is a structured report that makes the entire reasoning chain transparent: what data was used, what causal structure was assumed, which variables were adjusted for, what the effect estimate is, and what assumptions could break the analysis. This is designed to support expert judgment, not replace it — the human decides whether the assumptions are reasonable.

Framework Shift

Before (mainstream approach):        After (this paper):
                                     
  Domain expert                      Domain expert
      |                                  |
      v                                  v
  "I wonder if rain               "How does rain affect
   causes congestion?"              congestion during
      |                             peak hours?"
      v                                  |
  Must learn causal                  NL question
  inference tools,                       |
  manually specify DAGs,                 v
  choose adjustment sets,           Schema-aware LLM
  validate assumptions                 auto-translates
      |                                  |
      v                                  v
  Either gives up OR               PC algorithm learns
  makes mistakes OR                structure, bootstrap
  hires a statistician             validates, estimation
                                       |
                                       v
                                   Causal Card:
                                   estimate + assumptions
                                   + uncertainty + DAG

From requiring causal inference expertise to letting domain experts drive the investigation, the core shift is moving the burden of causal reasoning from the human user to the system while keeping humans in the loop for judgment.

Expert Assessment

Problem choice: This is a real gap. Traffic agencies genuinely have tons of observational data and lack the tools to do causal analysis. The intersection of “LLM accessibility” and “causal inference rigor” is underserved and practically important. It’s not a world-changing gap, but it’s a legitimate, useful one that sits at a productive intersection of several growing fields (LLM applications, causal ML, urban informatics).

Method maturity: The system is more of a thoughtful integration than a novel algorithmic contribution. The PC algorithm, bootstrap stability, DoWhy — none of these are new. The insight is in the pipeline design and the LLM-mediated query translation. This is clever engineering, not a breakthrough insight. A simpler approach — just wrapping DoWhy in a good UI with template queries — might get 80% of the value. But the schema-aware LLM translation and the structured Causal Card output add genuine usability beyond a simple wrapper.

Experimental integrity: The case studies on BDD-derived data are exploratory, not rigorous benchmarks. The authors are honest about this — the system is positioned as a hypothesis generation tool, not a source of definitive claims. However, the evaluation is thin: they show that the system produces “plausible” relationships, but there’s no ground-truth causal benchmark to measure against. How often does the PC algorithm produce wrong structures? How sensitive is the bootstrap stability to the number of resamples? These questions aren’t fully answered. No red flags, but the experimental section feels like a proof-of-concept rather than a validation.

Writing quality: The paper is clearly written with good structure. The Causal Card concept is well-motivated. The authors are admirably honest about limitations. The weakness is in the related work — a deeper comparison to existing causal inference UIs and LLM-based analysis tools would strengthen the positioning. The case studies could also benefit from more systematic exploration rather than cherry-picked examples.

Verdict: weak accept — A practical system at a useful intersection of LLMs and causal inference, honestly positioned as a hypothesis generation tool, but thin on rigorous evaluation and limited novelty in individual components.

Takeaways

1. The “Causal Card” concept transfers immediately. If you’re building any system that reports causal or quasi-causal findings to non-experts, the idea of packaging results with explicit assumptions, adjustment sets, and uncertainty is directly reusable. It turns a black-box answer into a transparent reasoning artifact.

2. Schema-aware LLM translation is a powerful pattern. When you need LLMs to translate natural language into structured queries (SQL, causal queries, API calls), providing the schema as context dramatically reduces hallucination. This applies far beyond causal inference — any “ask your database in English” system benefits from this approach.

3. Bootstrap stability for causal discovery is a cheap safeguard. If you’re applying the PC algorithm or any structure learning method to observational data, running bootstrap resampling to filter fragile edges costs almost nothing computationally and significantly improves reliability. This is a technique worth stealing for any causal analysis pipeline.

4. Positioning a system as “hypothesis generation, not definitive claims” is strategically smart. In causal inference, overclaiming is the cardinal sin. The authors’ honest framing protects them from criticism while still delivering practical value. This is a framing pattern worth emulating when your system has inherent epistemic limitations.