Tokenization Is a Change of Coordinates, Not a Bigger Dictionary

Paper: 2608.18025 Authors: Yi Wang Categories: cs.LG, cs.AI, cs.SD

The Gap

Symbolic music looks like the easiest possible transfer target for a GPT. It is already discrete, already sequential, already full of repetition and hierarchy and long-range dependency. So the field did the obvious thing: if language works because a finite vocabulary of reusable units recombines into unbounded structure, then find music’s reusable units. Music Transformer serialized MIDI events; REMI added bar and position symbols; compound tokens bundled attributes; MusicBERT and MuPT scaled it up; learned segment vocabularies like MuseTok went further and let a codebook discover chunks. Fradet et al. even ran BPE directly over symbolic music, which is the purest statement of the belief: merge frequent combinations, get shorter sequences, get better modeling.

The results have been fine but never language-like. The usual diagnoses are data scarcity, polyphony, or the fact that music has no semantics. This paper offers a different one, and it is a genuinely uncomfortable claim: the whole program confused a coding mechanism with the source of compression. Grouping several observations under one symbol shortens the sequence. It does not, by itself, make anything more predictable. Compression that helps a bounded model comes from placing facts in coordinates where their recurring regularities show up as stable conditional distributions. Chords and motifs are bigger boxes; they are not a better frame.

Once you say it that way, the research question inverts. Stop asking “what is the right musical unit?” Start asking “in what coordinate system do musical facts become predictively compressible, and at what point must the tokenizer stop and hand the rest to the model’s hidden state?”

[PROBLEM] GPT recipe transfers to music at the architecture
          level but not at the performance level; the field
          responds by hunting for bigger reusable units
          (chords, motifs, phrases, learned segments, BPE)
     |
     v
[DIAGNOSIS] "reusable combination" is a coding trick, not the
            source of compression.  a shorter sequence is not
            a shorter *predictive code*
     |
     v
[FRAMEWORK] Effectiveness--Losslessness.  two boundaries:
     |
     +-- Fact--Token   : decouple + denest until observable
     |                   regularities are exposed in the frame
     |
     +-- Token--State  : STOP before fixing context-dependent
                         relations; leave those to the state
     |
     v
[METHOD] coordinate-aware note = (type, pitch, duration)
         decoupled from a multi-scale musical-time coordinate
         derived from onset.  one token per original event.
         no chord / motif / voice / melody labels at all
     |
     v
[EVIDENCE] matched arms A..I on Pop-K, 3 seeds, one sealed test,
           metric = bits per *original event* (not per token)
             time coordinates:  3.167 -> 1.814   (-42.73%)
             BPE compaction:    -71.07% tokens, +14.00% bits
             fixed pitch geometry (chromatic / fifths): worse
             bijective pitch-class split: no measurable gain
     |
     v
[CONCLUSION] architectures transfer across modalities;
             tokenization interfaces do not

The Increment

One sentence: Before this paper, “better music tokenization” meant finding larger or smarter reusable units and was measured, implicitly, by sequence length; after it, the question is which reversible change of coordinates lowers predictive code length per observable event, with an explicit rule for where the tokenizer must stop meddling.

Core Mechanism

Start with the metric, because it is the part doing the real work. Define predictive code length as the negative log-likelihood a bounded model family can achieve over a representation, and — critically — normalize it by the number of original observable events, not by the number of tokens. This one normalization choice is what makes the whole comparison honest. Under bits-per-token, any merge scheme looks like a win, because you shrank the denominator. Under bits-per-event, a merge only wins if it actually made the music more predictable. Every arm in the paper is scored this way against a frozen learner (16 layers, width 64, about 817K parameters), same data split, same budget, three preregistered seeds, one sealed test read after checkpoints are locked.

The construction itself has two halves. The Fact–Token Boundary says which transformations should happen in the representation, and permits exactly two kinds. Decoupling separates factors the representation had glued together: an event is (type, onset, pitch, duration), and onset is position while the rest is content, so the token becomes a content triple plus a position coordinate rather than one joint identity. This is not an independence assumption — the model can still learn any content-position dependency it likes; it just no longer has to rediscover the same regularity separately for every position. Denesting makes known deterministic transformations explicit instead of leaving them buried inside learned computation. The flagship case is musical time. Token index is not musical time: simultaneous notes get different indices, adjacent notes can share an onset, and two metrically identical beats can sit arbitrarily far apart in serialization order. So onset is mapped to a coordinate carrying directed bar progress plus periodic within-beat, beat-within-bar, bar, four-bar and sixteen-bar phases, with signed onset, beat and bar distances entering attention as deterministic pairwise biases. The transform is invertible — the original onset is recoverable — it just stops hiding periodicity behind sequence position.

The Token–State Boundary says where to stop, and it is the half that is easy to get wrong. A contextual relation depends on surrounding context. If the tokenizer writes a fixed answer into the representation, it has replaced a whole family of context-dependent readings with one mapping fixed before context was ever available. So chord identity, motif membership, phrase structure, melody-versus-accompaniment, voice, and tonal function are all deliberately absent. All non-drum notes go into a single unsegregated stream with no track, part, or voice partition. The pitch experiments make the distinction concrete by testing three superficially similar operations: splitting pitch into pitch class plus register is bijective and merely reparameterizes; subtracting a known global transposition is reversible and removes a nuisance; projecting pitch class onto a chromatic circle or the circle of fifths looks like a nice inductive bias but silently commits to a fixed geometry of pitch proximity. Only the first two are legal under the framework, and the measurements agree.

  observable facts        coordinate construction        contextual state
  n = (type, onset,   ==> [ FACT--TOKEN BOUNDARY ]  ==> [ TOKEN--STATE ]
       pitch, dur)         decouple: content | pos      causal Transformer
                           denest:   tau = phi(onset)   resolves chord,
                                     multi-scale phase  motif, phrase,
                                     + pairwise bias    voice, function
                              |                              |
                              v                              v
                    z = ((type, pitch, dur), tau)     h = F(z_1..z_i, B)
                    one token per original event      non-invertible
                    NO chord/motif/voice labels       compression

  the three pitch operations, and which side of the line they fall on:

  +------------------------+-------------+---------------------------+
  | decomposition          | legal       | bijective; no gain shown  |
  |   p <-> (class, reg)   |             | (H: 1.8219 vs D: 1.8136)  |
  +------------------------+-------------+---------------------------+
  | canonicalization       | legal       | removes known nuisance    |
  |   p~ = p - s           |             | (F 3.0014 -> G 1.8138)    |
  +------------------------+-------------+---------------------------+
  | fixed relational       | ILLEGAL     | commits to a geometry     |
  | projection             |             | before context exists     |
  |   chromatic circle     |             | (E: 1.8905, worse)        |
  |   circle of fifths     |             | (I: 2.0425, worse)        |
  +------------------------+-------------+---------------------------+

  carrier coding is a separate axis, and it does not substitute:

     J  REMI-like        3.680 tok/event    2.4230 bits/event
     K  J + BPE          1.065 tok/event    2.7623 bits/event
     D  coordinate-aware 1.000 tok/event    1.8136 bits/event
                          ^^^^^              ^^^^^^
                    K wins on length,  D wins on what matters

The metaphor: Ptolemy versus Kepler. Both had the same planetary observations. Neither threw any away — both were, in this paper’s sense, lossless. Ptolemy wrote the positions in an Earth-centered frame, and in that frame the regularities are ugly: to predict where Mars goes next you need dozens of epicycles, and you must relearn the correction for each planet separately. Kepler wrote the same positions in a Sun-centered frame with the orbit as an ellipse, and suddenly one short law predicts all of them. No new data, no bigger model, no larger units. Just a change of coordinates that turned a mess of special cases into one stable conditional structure. That is exactly what the temporal coordinate does to musical time.

The metaphor keeps paying. Denesting is Kepler’s move: a transformation that was objectively true and reversible all along, pulled out of the fitting procedure and written into the frame. Carrier coding is what a scribe does when he abbreviates the star tables to save parchment — genuinely shorter, and it helps prediction not at all. That is BPE on music: 71% fewer symbols, 14% more predictive bits. And the Token–State Boundary is the mistake of writing “Mars is in retrograde” directly into your data table. Retrograde motion is real, but it is a relation between the planet and the observer’s position; freeze it into the coordinates and you have quietly destroyed the possibility of a frame in which it dissolves. Baking chord labels into music tokens is the same error, and the chromatic-circle arms are the measurable version of it: the geometry is not wrong, exactly, it is just fixed too early, and it costs bits.

Key Concepts

• Bits per observable event (versus bits per token): Suppose I compress a book by replacing every occurrence of “the” with a single symbol. My token count drops. Did I make the book more predictable? No — I just renamed things. If I now report “bits per token”, my number improves purely because each token carries more content, and I can keep gaming it by merging harder. The fix is to hold the denominator fixed at whatever the world actually contains — here, the original note and rest events. Then the only way to win is to genuinely lower the model’s surprise about the music. This is why the paper’s BPE control is so clean: K shortens the serialization by 71.07% and still needs 14.00% more bits per event than the stream it merged. Sequence compaction and predictive compression are simply different quantities, and the field has been reporting the wrong one.

• Decoupling is not an independence assumption: A common objection to splitting (type, onset, pitch, duration) into content plus position is “but pitch obviously depends on where you are in the bar.” True, and nothing in the decoupling denies it. The claim is narrower: when content and position share one joint symbol, the same content regularity has to be learned again at every position it appears in, because it wears a different name each time. Splitting them lets the model learn the regularity once and apply it everywhere, while leaving the joint dependency fully available through attention. The paper’s phrasing is precise — decoupling removes an interaction induced by the representation, not one present in the source.

• Relational losslessness: Ordinary losslessness asks whether you can reconstruct the input. That is not enough, because two exactly reconstructible representations can differ in what the model is still free to compute. If your tokenizer stamps every note with a chord label, you can still recover the notes — nothing is lost in the reconstruction sense — but you have collapsed a family of possible harmonic readings into one, chosen before any context was seen. The model can no longer entertain the alternative that this note group is a passing figure rather than a chord. Relational losslessness is the demand that context-dependent alternatives stay open until there is context available to settle them. It gives an unusually crisp answer to “how much should the tokenizer do?”: everything that is a deterministic consequence of the observations, and nothing that is a judgment call.

Framework Shift

Before (mainstream approach):        After (this paper):

 raw MIDI events                      raw MIDI events
      |                                    |
      v                                    v
 find bigger reusable units          rotate the frame
  chords / motifs / phrases          decouple content | position
  compound tokens                    denest onset -> multi-scale
  learned segment codebooks                 musical time
  BPE merges                         canonicalize away nuisance
      |                                    |
      v                                    v
 [ shorter sequence ]                [ same 1 token / event ]
      |                                    |
  measured by                         measured by
  tokens/event, vocab size            bits per ORIGINAL event
      |                                    |
      v                                    v
 structural labels baked in          STOP at the boundary:
  chord, voice, melody               chord/motif/phrase/voice
  track separation                   never enter the token
      |                                    |
      v                                    v
 model inherits the                  model computes them as
 tokenizer's musical                 contextual state; higher-
 opinions                            order organization emerges
                                     without any labels

From “find the right unit” to “find the right frame”, the core shift is that tokenization stops being a vocabulary-design problem and becomes a question of which reversible coordinate change lowers a bounded model’s predictive code — with an explicit stopping rule attached.

Expert Assessment

Problem choice: Excellent, and the framing earns its keep. The confusion between coding efficiency and predictive compressibility is real, it is widespread, and it is not confined to music — it is exactly the confusion behind “our tokenizer produces 30% fewer tokens, therefore it is better.” This sits naturally alongside Zouhar et al.’s noiseless-channel work and Schmidt et al.’s “tokenization is more than compression”, but it goes further by supplying a stopping criterion rather than just a quality measure. The Token–State Boundary in particular is the kind of idea I expect to see cited outside music: a clean statement of when an inductive bias is a gift and when it is a premature commitment.

Method maturity: The framework is more mature than the artifact. What actually gets built is, in the end, a note-level tokenizer with a good multi-scale metric-position encoding and relative temporal attention biases — and honestly, pieces of that already exist (Inaba et al. on time and pitch relativity, Guo et al.’s music embedding space, the whole relative-position literature). The contribution is not the encoding, it is the account of why that encoding beats bigger units, plus the discipline of testing the three pitch operations separately. That separation is the cleverest thing in the paper. The obvious simpler alternative the paper never confronts: give a large model absolute onset and enough data and let it learn the metric grid itself. At 817K parameters you can’t tell whether the coordinate is doing indispensable work or merely substituting for capacity.

Experimental integrity: Unusually disciplined in protocol, and unusually candid in Appendix H — preregistered seeds, lineage-aware splits so transposed duplicates cannot cross, a sealed test opened once, an honest note that one seed reverses the H-versus-D direction, and an explicit disclosure that an H/I run was resumed after an infrastructure failure. I wish more papers wrote that appendix. But three things temper the headline numbers.

First, and most importantly, the canonicalization result is close to circular. The authors apply a known random semitone shift to each example themselves, retain the shift, and then show that removing the shift they injected helps by 39.57%. That is a demonstration that removing an oracle-known nuisance is valuable — which nobody doubted. It says nothing about whether a system that must infer the key can capture that gain, and the paper is careful to say G infers no tonic or mode, but the abstract’s “reversible canonicalization improves predictive compressibility” reads much stronger than the experiment supports.

Second, the two corpora disagree in magnitude by two orders of magnitude of effect. On Pop-K the temporal coordinate buys 42.73%; on ComMU the same intervention buys 0.107 bits out of roughly 11.85, under 1%. The paper calls this “directional replication”, which is technically accurate and rhetorically generous. The gap between 42.73% and 0.9% is itself the most interesting unexplained number in the paper — Pop-K is an augmented corpus whose own creators note residual similarity between segments, and an absolute 1.81 bits per event is a very predictable corpus. I would want the Pop-K result reproduced on a non-augmented, non-pop corpus before treating 42.73% as the effect size.

Third, the emergence claim rests on two generated excerpts chosen by author listening out of ten candidates, with event Jaccard against the source of 25.8% and 50.8%. The paper discloses all of this, which is to its credit. But a continuation that shares half its exact (onset, pitch, duration) triples with the source is not strong evidence of emergent musical organization on an augmented dataset with known near-duplicates. This section proves existence at best, and the paper says so; the abstract’s “higher-order musical organization emerges” does not carry that caveat.

Writing quality: The theory sections are dense with defined terms — Predictive Effectiveness Principle, Relational Losslessness Principle, Fact–Token Boundary, Token–State Boundary, coordinate carrier — and the ratio of new vocabulary to new mechanism is high. Section 2 could be half its length with no loss, and would be far more persuasive if it opened with the BPE control, which is the paper’s single most convincing fact and is currently buried in section 5.3. The section I would rewrite is the abstract: it promises an answer to “why GPT-style models do not transfer” when the evidence supports a narrower and still-valuable claim — that under a small fixed learner on one pop-MIDI corpus, coordinate construction beats carrier coding. The title is writing a check the 817K-parameter experiments cannot cash.

Verdict: weak accept — the conceptual separation of coordinate construction from carrier coding is genuinely useful and the BPE control is a clean falsification of a widely held belief, but the headline effect sizes come from a small model on a near-duplicate-heavy corpus, and the independent replication quietly shrinks the effect to under one percent.

Takeaways

• Change your tokenizer metric today. Report bits per observable unit — per note, per character, per pixel, per whatever the world contains — never bits per token. Bits per token is gameable by merging and it has probably been flattering your tokenizer for years. This one line of the paper transfers to every modality and costs nothing to adopt.
• Sequence compaction and predictive compression are different quantities. The K arm is the fact to remember: 71% fewer tokens, 14% more predictive bits. Next time someone justifies a tokenizer change with a compression ratio, ask for the code length under a fixed learner.
• Sort your representation tricks into three bins before you run anything. Reversible reparameterization (probably neutral — the pitch-class split measured as no gain), nuisance removal (real win when the nuisance is knowable), and fixed relational projection (commits early, measurably costs you). Most “clever encoding” ideas are secretly in bin three. The circle of fifths is a beautiful, correct structure and it made things worse.
• Ask where your tokenizer stops making judgment calls. Anything the tokenizer decides is a decision the model can never revisit. Chord labels in music, sentence boundaries in text, object identity in video, entity linking in RAG chunking — if a downstream reading is context-dependent, freezing it in the representation is a permanent capacity loss, even when reconstruction is perfect. “Lossless” is not the right test; “did I leave the alternatives open” is.
• Serialization order is not time. The general lesson under the music specifics: whenever your sequence index is a serialization artifact rather than the domain’s real ordering variable, derive an explicit coordinate for the real one and feed it in — including as pairwise attention bias. This applies to event logs, irregularly sampled sensors, multi-track anything, and any domain where two things genuinely happen at once.
• Steal the protocol, not just the ideas. Preregistered seeds, lineage-aware splitting so augmented duplicates cannot straddle the split, a sealed test opened exactly once after checkpoint lock, and an appendix that lists what the evidence does not establish. That appendix is a better model for how to write limitations than most of what gets published.