Code-Switching for Machines: Splicing Object Crops Into Sentences to Fix Referential Ambiguity

Paper: 2608.11167 Authors: Changhao Xiang, Shangyu Xing, Zhen Wu, Jianbing Zhang, Xinyu Dai Categories: cs.CV, cs.CL, cs.LG

The Gap

The standard recipe for teaching an LLM to see is embarrassingly blunt. Take a frozen vision encoder, bolt on a projector, and train it on hundreds of thousands of image-caption pairs — LLaVA’s 558K/600K stage-one corpus is the canonical example, BLIP-2’s Q-Former is the same idea with more parameters in the bridge. The supervision signal is: here is a bag of visual tokens for the whole scene, here is a paragraph of prose, make them agree.

The problem is that “make them agree” hides an unsupervised assignment problem. A picture with a dog, a frisbee, and a patch of grass paired with the caption “a dog catches a frisbee on the grass” never tells the model which visual tokens correspond to which noun. The model has to recover that mapping from co-occurrence statistics across the whole corpus. That works — eventually, expensively — and it is why alignment pretraining is data-hungry and why grounding stays mediocre afterwards.

The grounding-focused line of work (Kosmos-2, Shikra, Ferret, GLaMM) noticed this and added location supervision, but almost all of it does so by serializing coordinates into text: “a dog [0.31, 0.44, 0.58, 0.79] catches a frisbee.” That is an indirection — the model reads numbers, then has to look up what lives at those numbers inside the global feature map. It reduces ambiguity without eliminating it, and it spends the language model’s capacity on learning a coordinate dialect.

MMCS asks a simpler question: if the caption says “dog,” why not put the dog there, in the sequence, instead of a word or a box?

PROBLEM: image-level alignment
   one global image representation <-> one long caption
   N visual objects, M text entities, zero links given
                    |
                    v
ASSUMPTION: the missing links are the bottleneck
   the model can solve the N x M assignment from
   corpus-wide co-occurrence, but only slowly
   ==> data inefficiency + weak grounding
                    |
                    v
METHOD: MultiModal Code-Switching
   replace the entity token with the object itself
   "a dog catches a frisbee"
     ->  "a <o1> catches a <o2>"
   + a synthesis pipeline for 773K such samples
                    |
                    v
EVIDENCE
   50K MMCS samples  >=  600K image-text pairs
   grounding + perception gains hold across model sizes
                    |
                    v
CONCLUSION
   the bottleneck was supervision *granularity*,
   not supervision *volume*

The Increment

One sentence: Before, object-level grounding had to be inferred from global image-text co-occurrence (or bolted on as textual coordinates); after, the correspondence is baked into the token sequence itself as an identity — the noun slot *is the object — and the alignment stage gets roughly an order of magnitude more data-efficient.

Core Mechanism

There are two halves: an offline data factory and a trivially simple training objective.

The factory inverts a grounding model. Start from an image-caption pair, parse the caption for entity nouns, and hand each noun to an open-vocabulary grounder to get a box. Then filter aggressively — and this filtering is the part that actually determines data quality. If a noun gets no box, drop it. If it gets several boxes (“two dogs”), the one-to-one correspondence is broken, so drop it or the whole sample. If the box is tiny, the crop carries no signal. What survives is a triple: image, caption, and a clean noun-to-region map. Repeat 773K times.

Assembly is where the “code-switching” happens. For each grounded noun, crop the region, push it through the same vision encoder and projector the model already uses, and substitute the resulting embedding into the text sequence at the noun’s position. The surrounding words — determiners, verbs, prepositions — stay as text. So the sequence keeps the grammar of language while its content words come from vision, exactly like a bilingual speaker dropping a Mandarin noun into an English sentence.

Then you train with plain next-token prediction on the text positions. This is the load-bearing detail: to predict “catches,” the model must have already extracted “this is a dog, dogs are agents that do dog things” from the object embedding at position i. The gradient has nowhere else to go. Compare this with a global-image setup, where the model can predict “catches” from language priors alone and never look at the pixels.

DATA SYNTHESIS (offline)
   image + caption
        |
        v
   [entity parser]  --> nouns: dog, frisbee, grass
        |
        v
   [open-vocab grounder]  --> candidate boxes per noun
        |
        v
   [filter]  -- drop: no box | multiple boxes | too small
        |         (only 1-to-1 pairs survive)
        v
   (image, caption, noun -> box) x 773K


SEQUENCE ASSEMBLY
   caption: "a  dog  catches a  frisbee  on the grass"
                 |                |
                 v                v
            crop(box)        crop(box)
                 |                |
          [vision encoder]  [vision encoder]
                 |                |
            [projector]      [projector]
                 |                |
                 v                v
   tokens:  a  <o1>  catches a  <o2>  on the grass
            ^          ^^^^^                ^^^^^
            text scaffolding preserved


TRAINING
   next-token loss on TEXT positions
   predicting "catches" requires reading <o1>
   ==> the object vector is pushed into lexical space

The metaphor: think of a recipe card in a professional kitchen.

The mainstream approach hands the line cook a photograph of the entire prep counter stapled to a paragraph of prose: “dice the shallot, then deglaze with the wine.” The cook can see everything and knows the words, but must figure out for himself which pale blob in the photo is the shallot and which is the garlic. Over a thousand shifts he’ll get it right by correlation. That’s 600K image-text pairs.

The coordinate-grounding approach improves the card: “dice the shallot (counter photo, grid ref B4).” Better — but now the cook has to learn a grid-reference dialect, and he still has to make the lookup, and a slightly-off grid reference lands him on the garlic.

MMCS staples the actual shallot to the card, in the slot where the word “shallot” would be. There is no lookup and no dialect. The recipe’s grammar — dice, then deglaze, sequence and syntax — stays in words, because that’s what words are good at; the referents are physically present. The data pipeline is the prep cook who takes the counter photo and pre-bags each ingredient; the filtering step is him refusing to bag anything when there are two shallots on the counter and he can’t tell which one the recipe meant. And the reason the trainee learns fast is that he can’t fake it: to know that this thing gets diced rather than deglazed, he has to actually look at the thing in the slot.

Key Concepts

• Referential ambiguity: The gap between “these two things are about each other” and “*this part goes with that word.” Show someone a photo of a crowded kitchen and say “the mise en place is ready” — they learn nothing about which object is a shallot. Show them a photo with three objects and a caption naming three objects, and there are six possible assignments; the caption doesn’t pick one. Contrastive and captioning objectives on whole images give you the first kind of signal and hope the second falls out. It does, at scale, which is precisely why the scale is needed.

• Code-switching, and why it’s the right analogy: In linguistics, code-switching isn’t random mixing — a bilingual speaker keeps one language’s grammatical frame and inserts the other language’s content words at points where they’re a cleaner fit (“我 sent 了 email”). The frame language handles structure; the embedded language handles the specific referent. MMCS takes exactly this division of labor: text keeps syntax, vision supplies nouns. The insight is that vision and language aren’t interchangeable descriptions of the same thing — they’re good at different jobs, and the switch point is where the alignment information lives.

• Substitution vs. appending: Nearly every interleaved-multimodal setup *appends — image, then text, then image, then text. Appending lets the model route around the visual input, because the text is self-sufficient. Substitution removes the textual crutch: the word is gone, so the object embedding is the only path to the answer. It’s the same trick as masked language modeling — you learn about a word by deleting it — applied across modalities. Cheap to implement, and it changes the gradient structure fundamentally.

Framework Shift

Before (LLaVA / BLIP-2 style):        After (MMCS):

  +--------------------+                +----+----+-------+
  |    whole image     |                | o1 | o2 |  o3   |
  +--------------------+                +----+----+-------+
            |                             |     |     |
     [global tokens]                    crop  crop  crop
            |                             |     |     |
            v                             v     v     v
  "a dog catches a frisbee           "a <o1> catches a
        on the grass"                  <o2> on the <o3>"
            |                                   |
            v                                   v
  model must GUESS which               slot position == the
  tokens mean "dog"                    correspondence itself
            |                                   |
            v                                   v
  needs ~600K pairs                    ~50K samples suffice
  to average out the noise             (per the paper)

From global-to-global matching to slot-level substitution, the core shift is that correspondence stops being something the model must infer and becomes something the data format asserts.

Expert Assessment

Problem choice: Real gap, and a well-chosen one. The data inefficiency of the alignment stage is a known embarrassment — everyone runs the LLaVA pretrain recipe and nobody defends it as principled. That said, the neighborhood is crowded: region-level pretraining (GLIP, RegionCLIP), grounded captioning (Kosmos-2), and referring MLLMs (Ferret, GLaMM, ASM) all attack the same ambiguity. MMCS’s claim to novelty is the *format — pixels substituted inline rather than coordinates appended — plus the pipeline that makes it scalable. That’s a real but incremental delta, and the honest framing is “a better format for the same insight,” not “a new insight.”

Method maturity: Clever rather than brute-force, and cheap enough to be adopted, which is the highest compliment for a pretraining trick. Three concerns. First, the pipeline inherits the grounder’s biases and errors wholesale — MMCS can only teach the model to align what GroundingDINO-class models already localize well, which is a ceiling nobody has measured. Second, deleting the noun from the text is a double-edged sword: the model may learn object-vector-to-context mapping while *weakening the word-to-object link, unless normal captions are mixed in; I’d want to see that ablation. Third, cropped-object embeddings are context-free by construction, so the model sees a dog with no scene around it — this may help identity and hurt spatial relations, and the paper’s own emphasis on grounding benchmarks makes this worth checking carefully.

Experimental integrity: The headline “50K beats 600K” is rhetorically strong but the comparison deserves scrutiny. It’s a comparison against the LLaVA-style pretrain corpus, which is a weak baseline *specifically on grounding — of course object-level supervision wins on RefCOCO-style tasks. The sharper question is whether the gains survive full instruction tuning, since alignment-stage advantages notoriously wash out once you run a large SFT stage on top. There’s also a distributional-proximity worry: the synthesis pipeline uses a grounder trained on data close to the grounding benchmarks, so some of the improvement may be distillation from that grounder rather than better alignment per se. And a small tell: if 50K matches 600K, why build 773K? Either the curve saturates early (in which case say so and show it) or the 773K results are less flattering than the 50K headline. The scaling curve is the most load-bearing figure in the paper and it’s the one the abstract avoids quantifying. Everything here is judged from the abstract’s claims; the tables would settle it.

Writing quality: The framing is excellent — “code-switching” is a genuinely illuminating name and does real explanatory work, which is rarer than it should be. The corner most likely cut is the pipeline section: filtering heuristics, multi-instance handling, and crop-size policy are where all the quality lives and where papers like this typically compress into two sentences and a supplementary table. Rewriting the data-scaling analysis — 50K vs. 200K vs. 773K, with and without normal captions mixed in — would elevate this from “nice trick with a good name” to “recipe others adopt.”

Verdict: weak accept — a well-motivated, cheap, reproducible format change with a compelling data-efficiency claim, held back by a favorable baseline choice and an under-examined dependence on the grounder that generated its data.

Takeaways

• Substitute, don’t append. If you want a model to actually use a modality, delete the redundant modality rather than adding to it. Appending permits shortcuts; substitution closes them. This transfers directly: splice an audio clip where the speaker’s name goes, a table cell where the number goes, a function body where the identifier goes.
• Invert a specialist model to synthesize supervision. You don’t need human annotation for object-entity links — a grounder plus aggressive filtering gives you 773K samples. The general pattern: whenever you want fine-grained alignment labels, find an off-the-shelf model that produces one direction of the mapping and filter hard for the unambiguous cases. Be aware you’re distilling that model’s competence and its blind spots.
• The filter is the method. “Keep only one-to-one correspondences” is doing more work here than the architecture. When you build synthetic data, ambiguity-rejection rules deserve as much design attention as the generation step.
• Reframe volume problems as granularity problems. The reflex when alignment is weak is to add data. Ask instead what the current format fails to state explicitly, and whether stating it directly changes the gradient structure. That question is worth asking about most pretraining corpora.
• Steal the linguistic framing. The frame-language/embedded-language split — one modality supplies structure, another supplies referents — is a useful design axis for any multimodal sequence format, and it’s more principled than “interleave everything.”