Triadic Suffix Tokenization: Teaching LLMs to Count by Giving Numbers a Backbone

Paper: 2604.11582 Authors: Olga Chetverina Categories: cs.CL, cs.AI, cs.LG

The Gap

Standard subword tokenizers (BPE, WordPiece, SentencePiece) treat numbers as just another text stream. They fragment “123456” inconsistently — sometimes as “123|456”, sometimes as “12|34|56”, depending on training corpus statistics. This destroys two critical properties: positional structure (which digit is in the thousands place?) and decimal alignment (where’s the decimal point relative to each digit?). The result: LLMs fail at basic arithmetic not because they can’t learn addition, but because the input representation scrambles the problem before the model even sees it.

Prior fixes fall into two camps: (1) character-level tokenization (preserves structure but explodes sequence length), (2) learned embeddings that try to recover position post-tokenization (adds parameters and training complexity). Both treat the symptom, not the cause. The cause is that tokenization throws away the very structure arithmetic depends on.

Problem: Standard tokenizers fragment numbers inconsistently
   |
   v
Assumption: If we chunk deterministically + annotate magnitude explicitly
   |
   v
Method: Triadic Suffix Tokenization (3-digit groups + magnitude markers)
   |
   v
Evidence: [Deferred - no experiments yet]
   |
   v
Conclusion: Should provide stable gradient signal for numerical reasoning

The Increment

One sentence: Before this paper, numbers entered LLMs as structurally ambiguous token sequences; after, they arrive as magnitude-annotated triads with explicit positional semantics baked into the vocabulary.

Core Mechanism

TST operates in three stages. First, it partitions any number into three-digit groups (triads) from the decimal point outward in both directions. For 1234567.89, that’s [1][234][567].[890]. Second, it assigns each triad a magnitude suffix: the integer side gets “thousand”, “million”, “billion” markers; the fractional side gets “thousandth”, “millionth” markers. Third, it either (a) creates fixed vocabulary entries like “234_thousand” covering all 1000 possibilities per magnitude, or (b) uses dynamic suffix tokens like “234” + <MAG_3> that compose at runtime.

The vocabulary-based variant pre-allocates 10,000 tokens (1000 triads × 10 magnitudes) to cover 10^-15 to 10^18. The suffix-marker variant uses ~20 special tokens that attach to any digit sequence. Both preserve exact digits while making magnitude relationships explicit: “234_million” and “567_thousand” carry their relative scale in the token itself, not as an inference problem for the model.

Input: 1234567.89
   |
   v
[Partition into triads from decimal point]
   |
   +---> Integer side: 1 | 234 | 567
   |
   +---> Fractional side: 890
   |
   v
[Assign magnitude markers]
   |
   +---> 1_million | 234_thousand | 567_unit | 890_thousandth
   |
   v
[Tokenize using fixed vocab OR dynamic suffixes]
   |
   v
Output: [TOK_1M] [TOK_234K] [TOK_567] [TOK_890m]

Think of it like a postal address system. Standard tokenizers are like writing “1234 Main Street” but sometimes splitting it as “12|34 Main” and sometimes as “123|4 Main” — the mail carrier (model) has to guess where the house number ends. TST is like requiring “1234 Main St, Block 12, District 3” — every component has an explicit label. The number “234” isn’t just three digits; it’s “234 in the thousands place”, which is a different entity from “234 in the millions place”. The model doesn’t have to infer position from context; position is part of the token’s identity. When you add “234_thousand” + “567_unit”, the magnitude markers tell you exactly how to align them, just like “Block 12” + “Block 13” tells you they’re neighbors.

Key Concepts

• Magnitude marker: A suffix that explicitly encodes a digit group’s order of magnitude. Instead of letting the model infer that “234” in “1234567” represents 234,000, TST makes it “234_thousand”. This is like the difference between showing someone a ruler with unlabeled tick marks versus one where every centimeter is numbered — the second one doesn’t require you to count from zero every time. For fractional parts, it works in reverse: “890” after a decimal becomes “890_thousandth”, making it clear we’re talking about 0.890, not 890. The key insight: magnitude is a property of position, and position should be explicit, not implicit.

• Triadic chunking: Grouping digits in threes from the decimal point outward. Why three? Because human numerical notation already uses this (1,234,567), and it aligns with named magnitudes (thousand, million, billion). But the deeper reason: three digits give you 1000 unique patterns, which is small enough to enumerate in a vocabulary but large enough to be efficient. Two digits would require more tokens for the same range; four would bloat the vocabulary. It’s the Goldilocks zone where vocabulary size and coverage balance. Each triad becomes an atomic unit — “234” is one token, not three separate digits fighting for attention in the embedding space.

Framework Shift

Before (standard subword):           After (TST):

Number: 1234567                      Number: 1234567
   |                                    |
   v                                    v
[BPE fragments statistically]        [Deterministic triadic split]
   |                                    |
"12|34|56|7"  OR  "123|456|7"        "1|234|567"
   |                                    |
   v                                    v
[Model infers position]              [Explicit magnitude markers]
   |                                    |
Embedding space:                     Token identities:
[12] [34] [56] [7]                   [1_million] [234_thousand] [567_unit]
  ^    ^    ^   ^                       ^            ^              ^
  |    |    |   |                       |            |              |
Position is implicit                 Position is explicit

From statistical fragmentation to deterministic structure, the core shift is: numbers stop being text and start being annotated quantities.

Expert Assessment

Problem choice: Real gap, not manufactured. LLMs’ arithmetic failures are well-documented, and tokenization is a plausible culprit — it’s the first place structure gets lost. The problem sits at the intersection of representation learning and numerical reasoning, which is timely given the push for LLMs to handle scientific/financial tasks. However, the paper oversells the novelty slightly; digit grouping and magnitude annotation aren’t new ideas (see: scientific notation, financial formatting). The contribution is formalizing this for LLM tokenization.

Method maturity: Clever insight, not brute force. The triadic structure is elegant because it piggybacks on existing human conventions. But there’s a simpler approach being overlooked: just use scientific notation (1.234567e6) and tokenize that. The paper doesn’t compare against this baseline or explain why triads are superior. The two variants (fixed vocab vs. dynamic suffixes) feel underexplored — no analysis of which is better when, or what the trade-offs are beyond vocabulary size.

Experimental integrity: No experiments. The paper explicitly defers validation to future work, which is a massive red flag. Without empirical evidence, we’re evaluating a hypothesis, not a result. The claims about “stable convergence” and “consistent gradient signal” are plausible but unproven. This reads like a position paper or a preprint that escaped too early.

Writing quality: The abstract and method description are crisp, but the paper lacks depth. Section 3 (the method) is only a few paragraphs — it should be twice as long, with worked examples, edge case handling (what about scientific notation inputs? negative numbers? leading zeros?), and complexity analysis. The related work section is missing entirely, which makes it hard to position this against prior art. If the authors rewrote Section 4 (experiments) with even toy results — training a small model on arithmetic tasks with TST vs. standard tokenization — the paper would jump from “interesting idea” to “credible contribution”.

Verdict: weak reject — Promising idea undermined by zero experimental validation; feels like a workshop paper submitted to a main venue.

Takeaways

The core transferable idea: when your model struggles with structured data, check if your tokenization is destroying the structure before the model sees it. This applies beyond numbers — think dates (YYYY-MM-DD), coordinates (lat/lon), chemical formulas (H2O), version numbers (1.2.3). The pattern: if humans use positional or hierarchical conventions to encode meaning, your tokenizer should respect those conventions, not treat them as arbitrary text.

Concretely, practitioners can steal the “annotate magnitude explicitly” principle. If you’re fine-tuning on financial data, don’t let BPE fragment “$1,234,567” inconsistently — preprocess it into magnitude-tagged chunks. If you’re working with scientific measurements, don’t let “1.23e-5” get split randomly — tokenize the mantissa and exponent separately with explicit markers. The vocabulary-based approach (pre-allocating tokens for common patterns) is a useful trick when you have a bounded domain and want deterministic behavior.

The deeper lesson: tokenization is not a solved problem. We’ve been using subword methods designed for natural language on data that isn’t natural language. Numbers, code, structured formats — these need domain-specific tokenization strategies. TST is one example; the field needs more.