FAME: Pinpointing Which Log Line Failed Using Mixture-of-Experts

Paper: 2605.22779 Authors: Huanchi Wang, Zihang Huang, Yifang Tian, Kristina Dzeparoska, Hans-Arno Jacobsen, Alberto Leon-Garcia Categories: cs.SE, cs.LG

The Gap

Existing log anomaly detectors work at session or window level—they flag a batch of 50-200 log lines and tell you “something’s wrong in here.” Operators then manually grep through routine messages to find the actual culprit. Message-level detection would pinpoint the exact line, but three obstacles block the path: (1) the same template (“disk read failed”) appears in both normal retries and catastrophic failures, (2) failures come from heterogeneous subsystems (network, storage, compute) requiring different reasoning, and (3) labeling millions of individual lines is prohibitively expensive. Recent LLM-based detectors can reason over log semantics but cost too much to run on every line in production.

Problem: Session-level detection buries signal in noise
   |
   v
Assumption: Message-level needs semantic reasoning + subsystem awareness
   |
   v
Method: Use LLM once offline to partition templates into failure domains,
        then train lightweight domain experts for online inference
   |
   v
Evidence: BGL F1=98.16 with K=100 labels (76x reduction), 86.3% unseen detection
   |
   v
Conclusion: Offline LLM guidance + online lightweight experts = practical message-level detection

The Increment

One sentence: Before FAME, you either paid LLM costs per line or settled for coarse session-level alerts; after FAME, you get message-level precision with lightweight models by using the LLM once to design the expert architecture.

Core Mechanism

FAME operates in two phases. Offline, you sample at most K lines per event template (e.g., “Node <*> failed”), feed them to an LLM with failure context, and get back binary labels (normal/anomaly) plus representative examples. The LLM then proposes a partition of all templates into failure domains—clusters like “network errors,” “disk failures,” “memory issues.” A certification step validates this partition: if templates in the same domain have inconsistent labels or semantics, the partition is rejected and the LLM tries again.

Once certified, FAME trains a router and domain-specific expert models. The router is a lightweight classifier that reads a log line’s embedding and predicts which failure domain it belongs to. Each expert is a small neural network specialized for one domain. At inference time, a new log line hits the router, gets routed to the appropriate expert, and receives both an anomaly score and a failure-domain label. The entire pipeline runs on-premise with no LLM calls.

Offline Phase:
  Templates --> Sample K lines --> LLM annotates --> Binary labels + examples
       |                                                      |
       +------------------------------------------------------+
       |
       v
  LLM proposes domain partition --> Certification (validate consistency)
       |
       v
  Train router + domain experts

Online Phase:
  Log line --> Embedding --> Router --> Expert_domain --> [Anomaly score, Domain label]

Think of FAME as a hospital triage system designed by a veteran doctor. The doctor (LLM) examines a few cases from each symptom type, writes diagnostic guidelines, and organizes symptoms into departments (cardiology, neurology, orthopedics). Then junior doctors (lightweight experts) staff each department. When a patient arrives, a nurse (router) reads their chart and sends them to the right department. The veteran doctor never sees patients again—their expertise is baked into the department structure and guidelines. If the department assignments don’t make clinical sense (e.g., chest pain and broken bones in the same unit), the veteran redesigns the layout before opening the hospital.

Key Concepts

• Event Template: A log line like “Disk read error on /dev/sda1 at 14:32

  ” gets parsed into a template “Disk read error on <**> at <*>” where <*> marks variable fields. The template captures the message type; the variables hold instance-specific data. The same template can appear in both normal operations (transient retry) and failures (disk dying). This ambiguity is why session-level detectors struggle—they can’t distinguish context from structure alone.

• Failure Domain: A semantic cluster of templates that share root causes or subsystem origins. For example, templates like “Network timeout,” “Connection refused,” and “Packet loss detected” belong to a “network” domain, while “Out of memory,” “Malloc failed,” and “Swap exhausted” form a “memory” domain. The key insight: anomalies within a domain follow similar patterns, so a specialized expert can learn them efficiently. Cross-domain experts would dilute signal with irrelevant features.

• Certification Step: After the LLM proposes a domain partition, FAME checks whether templates in each domain have consistent labels and semantics. If domain D contains both mostly-normal and mostly-anomalous templates, or if templates have unrelated failure modes, certification fails. The LLM must revise the partition. This prevents the router from learning meaningless groupings and ensures each expert has a coherent learning task. It’s a sanity check before committing compute to training.

Framework Shift

Before (session-level detection):        After (FAME):
                                         
  [Log lines] --> Window aggregation     [Log line] --> Router --> Expert_net
      |              |                        |              |
      v              v                        v              v
  Feature vector --> Detector            Embedding      Anomaly score
      |                                       +
      v                                   Domain label
  Session-level alert                        
  (inspect 50-200 lines manually)        Message-level alert
                                         (exact line identified)

From batch-level flags to message-level precision, the core shift is using an LLM once to architect the detector rather than running it continuously.

Expert Assessment

Problem choice: Real gap. Session-level detection genuinely frustrates operators—I’ve watched SREs grep through hundreds of lines per alert. Message-level detection is a natural next step, and the label-efficiency angle addresses the practical blocker (annotation cost). The problem sits at the intersection of systems reliability and ML efficiency, which is timely given LLM deployment costs.

Method maturity: Clever use of LLM as a one-time architect rather than a runtime component. The certification step is the key innovation—without it, the LLM’s partition could be arbitrary. The mixture-of-experts framing is standard, but the offline LLM guidance + online lightweight inference is a smart tradeoff. One concern: the method assumes failure domains are discoverable from templates alone. In practice, some anomalies emerge from interaction patterns across subsystems, which template clustering might miss.

Experimental integrity: Baselines are fair (DeepLog, LogRobust, PLELog). The K=100 annotation budget is realistic. BGL and Thunderbird are standard benchmarks, though both are from HPC systems—generalization to cloud microservices or edge devices is unproven. The 86.3% unseen EventID detection is impressive but raises a question: what characterizes the 13.7% that fail? The paper doesn’t analyze failure modes. The Thunderbird result (F1=99.95, perfect recall) seems almost too good—worth checking if the dataset has label leakage or is unusually clean.

Writing quality: The abstract and intro are crisp. Section 3 (method) buries the certification step in a subsection—it should be elevated since it’s the core contribution. The related work section is thorough but reads like a literature dump; cutting it by 30% would improve flow. Figure 2 (architecture diagram) is clear, but Figure 4 (certification example) is hard to parse without reading the caption three times.

Verdict: weak accept — Solves a real problem with a practical method, but experimental scope is narrow (HPC logs only) and the certification step needs deeper analysis of when it succeeds vs. fails.

Takeaways

Steal the offline-online split: Use an expensive model once to design the architecture (partition the problem, generate training data, propose structure), then train cheap models for deployment. This pattern applies beyond logs—think network traffic classification, sensor anomaly detection, or document routing.

Certification as a design primitive: Before training a multi-model system, validate that the problem decomposition makes sense. If your router sends semantically unrelated inputs to the same expert, you’re training on noise. Add a programmatic check that rejects bad partitions early.

K-shot annotation per template: Instead of labeling uniformly across the dataset, sample K examples per category (template, user segment, event type). This gives you coverage of the decision boundary without drowning in redundant labels. Works when within-category variance is low.