JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management

Paper: 2609.17475 Authors: Yuhua Chen Categories: cs.AI, cs.PF

The Gap

Capable open-weight models make local coding and reasoning attractive: no cloud round-trip, no data leaves the machine, latency is yours. The blocker is not the weights alone. Context and execution state strain laptop memory.

That is the honest bottleneck. A quantized 27B model can fit; a long coding session, a full repo map, or an extended reasoning trace may not. Existing runtimes either cap context well below the model’s advertised window, thrash when state grows, or lose state across transitions.

The paper’s framing of the gap is systems-correct: you need mechanisms that fuse reconstruction and coordinate just-in-time materialization and release, independently of model-weight quantization. Quantizing weights harder is not the same as managing KV and component residency.

   THE LOCAL SERVING BOTTLENECK

   capable open-weight models
     local coding + reasoning is attractive
     no cloud round-trip, data stays on device
         |
         v
   blocker is not weights alone
     CONTEXT + EXECUTION STATE strain laptop memory
         |
         v
   a quantized 27B may FIT
   a long session / repo map / long reasoning trace may NOT
         |
         v
   existing runtimes
     cap context far below advertised window
     OR thrash as state grows
     OR lose state across transitions
         |
         v
   what is actually needed
     fuse reconstruction
     coordinate just-in-time materialization AND release
     independent of weight quantization
     (harder weight quant != state management)

The Increment

One sentence: Before JustFit, a 24 GiB laptop serving a 27B open-weight model completed about 30K context positions; after it, compressed KV execution plus residency and state-preserving transitions complete 212,992 positions — 6.93x — without giving up AIME-level reasoning.

Core Mechanism

Three mechanisms, one runtime (MLX-based).

KVExec: compressed KV execution. Rather than keeping a full-precision, fully materialized key-value cache for the entire context, JustFit compresses and executes against a managed KV state. The abstract is careful that this is independent of model-weight quantization — the weights can already be MXFP4; the win is in the activation/state path.

PhaseSwap: component residency. Not every runtime component needs to be resident at every moment. PhaseSwap decides what stays in memory by phase, trading compute for footprint in a coordinated way.

StateTrans: state-preserving serving transitions. This is the piece that turns “can run once” into “can serve.” Transitions between modes (or phases of a request) preserve state instead of discarding and rebuilding it, which is what usually collapses long sessions.

These mechanisms fuse reconstruction and coordinate just-in-time materialization and release. The design principle is materialize what you need when you need it, release it when you do not, and never lose the logical session across that churn.

   JustFit = KVExec + PhaseSwap + StateTrans

   [KVExec] compressed KV execution
     do not keep full-precision full-length KV always resident
     execute against a managed/compressed KV state
     independent of weight quantization
     (weights may already be MXFP4)
         |
         v
   [PhaseSwap] component residency
     not every component resident in every phase
     decide residency by phase
     trade compute for footprint in a coordinated way
         |
         v
   [StateTrans] state-preserving transitions
     mode/phase transitions PRESERVE state
     do not discard-and-rebuild
     this is what usually kills long sessions
         |
         v
   design principle
     materialize what you need WHEN you need it
     release when you do not
     never lose the logical session across that churn

The headline capacity test is unusually concrete. Full-execution capacity tests on a 24 GiB M4 Pro MacBook running Qwen3.8-27B MXFP4: three independent runs complete 196,608 input and 16,384 output tokens, increasing completed single-request context from the mlx-vlm baseline’s 30,720 positions to 212,992 (6.93x). A separate two-request run retains 229,376 positions in aggregate.

Performance is reported separately, which is the right hygiene: a 32K-input, 64-output probe reaches 19.11 tokens/s, and a repeated 32K+6K workload has a median peak process footprint of 16,374 MiB. That peak is what makes the 24 GiB claim credible rather than promotional.

And the quality check that prevents a pure systems paper from becoming a “long context that forgets” paper: the integrated runtime answers 29 of 30 AIME 2026 problems correctly, showing compact state and lifetime-aware execution still support extended generated reasoning.

   MEASURED RESULTS

   hardware: 24 GiB M4 Pro MacBook
   model:    Qwen3.8-27B MXFP4
   baseline: mlx-vlm

   capacity (full execution, 3 independent runs)
     completed single-request context
       baseline:  30,720 positions
       JustFit:   212,992 positions   = 6.93x
     work completed: 196,608 input + 16,384 output tokens
     two-request run: 229,376 positions aggregate

   performance (separate tests)
     32K in / 64 out probe: 19.11 tokens/s
     repeated 32K+6K workload:
       median peak process footprint 16,374 MiB
       (this is what makes "fits in 24 GiB" credible)

   quality
     AIME 2026: 29 / 30 correct
     long state did not mean broken reasoning

Think of it as a chef with a small counter who stops laying out every ingredient at once. The recipe (model) is unchanged. What changes is that ingredients (KV, components) are pulled from the pantry just in time, used, and returned — and the dish (the logical session) is never thrown away when the counter is cleared. The AIME score is the taste test: the meal still has to be good.

Key Concepts

• State, not just weights, is the laptop bottleneck: quantizing the model does not manage the KV cache or component residency.
• Just-in-time materialization and release: coordinate bring-in and tear-down so peak footprint is what matters, not theoretical maximum.
• Preserve state across transitions: discard-and-rebuild is what collapses serving; StateTrans is the fix.
• Separate capacity from performance from quality: 6.93x context, 19 tok/s, 29/30 AIME are three different claims, correctly measured separately.
• Independence from weight quantization: the techniques compose with MXFP4 rather than replacing it.

Framework Shift

Before (local serving limited by resident state):
  quantized weights fit on a laptop
  -> full KV + components still blow the budget
  -> completed context ~30K on 24 GiB
  -> transitions drop state; long sessions fail

After (just-in-time state management):
  KVExec + PhaseSwap + StateTrans
  -> completed context ~213K (6.93x)
  -> peak process footprint ~16.4 GiB on a 24 GiB machine
  -> transitions preserve the logical session
  -> AIME 29/30: long state still reasons

From “local LLM means a small model and a short window,” to “local LLM means aggressive, quality-checked state lifetime management on the machine you already own,” the core shift is that context length stops being a fixed property of the weights and becomes a property of the runtime.

Expert Assessment

Problem choice: Excellent and under-discussed. The open-weight community celebrates parameter counts and quantization formats; the real ceiling on a laptop is often state. Naming KV execution, residency, and transitions as separate mechanisms is the right systems decomposition.

Method maturity: Three named mechanisms with a fused design principle is good architecture talk. The abstract is a systems contribution: it will live or die on implementation detail, interaction with MLX kernels, and whether the compression is quality-neutral across tasks. The AIME result is a strong attempt to answer the obvious quality objection, but one benchmark family is not a general quality guarantee — long-context retrieval and agentic tool traces would be the next natural stress tests.

Experimental integrity: Strong for a systems abstract. Three independent capacity runs; baseline named (mlx-vlm); a separate two-request aggregate; performance reported apart from capacity; median peak process footprint rather than a vanity average; and a quality end-check. The 6.93x is credible because the baseline and the peak memory are both stated. Residual questions: how much of the win is compression vs residency policy; thermal/throttling on sustained laptop load; behavior past the measured ~213K.

Writing quality: Numbers are specific (positions, tokens/s, MiB, AIME count) and the independence-from-weight-quant claim is stated early, which prevents a common misreading. 13 pages, 4 figures, 9 tables is a reasonable packing for this claim set.

Verdict: weak accept — a concrete, well-instrumented local-serving result that treats state lifetime as the product, with quality checked rather than assumed. Not a universal MLX silver bullet, but exactly the kind of systems work laptop inference needs.

Takeaways

• When weights already fit, profile state. KV and component residency are often the real ceiling.
• Materialize and release just in time; measure peak footprint, not average comfort.
• Preserve logical session state across transitions. Serving breaks at the seams, not in the forward pass.
• Systems papers that only measure capacity are incomplete. Check that the model still reasons after the memory tricks.