The Illusion of Cross-Lingual Safety: Why Refusal Doesn't Travel to Low-Resource Languages

Paper: 2608.11146 Authors: Abigail Oppong, P Sam Sahil, Tadesse Destaw Belay, Maryam Ibrahim Mukhtar, Esmael Ahmed Abdu, Tassallah Abdullahi, Jessica Oparebea, Saminu Mohammad Aliyu, Idris Abdulmumin, Abubakar Juma Chilala Categories: cs.CL

The Gap

The multilingual jailbreak literature already established the surface fact: translate a harmful prompt into Zulu or Scots Gaelic and safety-tuned models often comply. Yong et al.’s low-resource jailbreak work, MultiJail, XSafety and the various translated-AdvBench derivatives all documented this. But nearly all of that work is generation-based — you prompt, you read the output, you classify it as refusal or compliance. That measurement strategy has two structural weaknesses that this paper targets.

First, generation-based evaluation confounds safety failure with capability failure. When a model produces garbled Twi in response to a harmful Twi prompt, is that a refusal? A comply? An artifact of the model not really speaking Twi? Judge models — usually GPT-4-class, themselves weak in these languages — are asked to arbitrate, and their unreliability propagates silently into the headline numbers. Second, generation-only evidence cannot distinguish two very different internal stories: (a) the model doesn’t understand the harmful request in Hausa, versus (b) the model understands it perfectly well but the understanding never reaches whatever circuitry produces refusals. These have opposite engineering implications — (a) says fix the tokenizer and pretraining mix, (b) says fix the alignment routing — and the field has been treating them as one problem.

There’s a second gap on the data side. Existing multilingual safety benchmarks are overwhelmingly translations of English harm taxonomies. That bakes in an assumption that harm is culturally invariant, which is exactly what you should not assume when the whole paper is about whether safety generalizes. A literal translation of “how do I buy a gun without a background check” may be near-meaningless in a jurisdiction with different institutions, while genuinely dangerous local prompts — around traditional medicine, ethnic tension, local financial scams — simply do not exist in translated benchmarks.

LoDNA plus the latent geometric probe is the paper’s response to both: paired literal/localized prompts to separate translation artifacts from cultural specificity, and hidden-state geometry to separate comprehension from refusal routing.

  PROBLEM
  [safety alignment trained in English]
        |
        v
  IMPLICIT ASSUMPTION
  [there exists one language-agnostic
   "harm manifold"; refusal generalizes]
        |
        +---- prior tests: generate + judge ----> confounded
        |     (capability failure ~= safety failure?)
        |
        v
  METHOD (this paper)
  [LoDNA: literal || localized pairs]  x  [4 langs: tw ha am sw]
        |
        v
  [extract hidden states per layer]
        |
        v
  [refusal direction r = mean(harmful) - mean(benign), in English]
        |
        v
  [project non-English prompts onto r  ==>  retained refusal signal]
        |
        v
  EVIDENCE
  * < 10% of English refusal signal retained (most lang-model pairs)
  * literal vs localized: cos 0.95 - 0.996  (semantically same)
  * but layerwise drift between them grows with depth
        |
        v
  CONCLUSION
  [meaning IS encoded; it is NOT routed to safety.
   multilingual alignment is superficial, not universal]

The Increment

One sentence: Before, we knew low-resource-language prompts *elicit unsafe outputs; after, we have hidden-state evidence that the models actually encode the harmful meaning and simply fail to connect it to the refusal mechanism — which relocates the bug from comprehension to routing.

Core Mechanism

The pipeline has three stages. Stage one is dataset construction. For each harmful concept, LoDNA holds two versions in each of Twi, Hausa, Amharic and Swahili: a literal translation of an English seed prompt, and a culturally localized prompt expressing comparable harmful intent in terms that make sense in the local context — local institutions, local substances, local social fault lines. Both are authored or validated by native speakers (the author list is largely African NLP researchers, which is what makes this dataset credible rather than MT output). A benign control set mirrors the same languages and surface style, which matters because you need a contrast set to define what “refusal” even looks like geometrically.

Stage two extracts the refusal direction. Run the English harmful set and the English benign set through the model, capture the residual-stream hidden state at each layer (typically at the final prompt token), and take the difference of means: r_L = mean(h_L | harmful_en) - mean(h_L | benign_en). This is the well-known difference-in-means refusal direction from the mechanistic-interpretability literature — a single vector per layer that, when ablated, reliably breaks refusal behavior in English. It functions here as a ruler: it defines the axis along which “I should refuse this” is written inside the model.

Stage three is the measurement. Push the Twi/Hausa/Amharic/Swahili prompts through the same model, grab the same layers, and project onto r_L. The retained fraction — how much of the English harmful-prompt projection magnitude survives in the non-English case — is the headline metric, and it comes out under 10% for most language-model pairs. Separately, cosine similarity between the literal and localized versions of the same harmful concept is computed layerwise. Early layers: 0.95–0.996, essentially identical representations. Deeper layers: the two drift apart. That combination is the argument. High similarity says the model is not confused about semantics. Low projection says that semantic content never lands on the refusal axis. Growing drift says whatever downstream processing exists treats literal and localized harm as different objects, so patching one form won’t cover the other.

  [LoDNA]
    |
    +-- EN seed harmful ---+
    +-- EN benign ---------+
    |                      |
    |                  (calibration)
    |                      |
    |                      v
    |            +---------------------------+
    |            | model forward pass        |
    |            | capture h at layers 1..L  |
    |            +---------------------------+
    |                      |
    |                      v
    |        r_L = mean(h|harm_en) - mean(h|benign_en)
    |                      |
    |                      |   * the "refusal ruler" per layer
    |                      |
    +-- tw / ha / am / sw --+
        (literal, localized)|
                            v
                +-----------------------------+
                | project h_nonEN onto r_L    |
                +-----------------------------+
                    |                     |
                    v                     v
          retained signal < 10%     cos(literal, localized)
                                     0.95 .. 0.996 early
                                       \
                                        \  drift grows
                                         v  with depth
                                    layer 1 ...... layer L
                                    [ ==== ] ... [ =   = ]
                                     tight        split

Structural metaphor: an airport with one working X-ray lane.

Think of the model as an airport terminal. Passengers are prompts. Somewhere deep in the building there is a security checkpoint — the refusal mechanism — and it works well: put an English passenger carrying a weapon through it, and the alarm fires. The refusal direction r_L is literally the alarm wire: the paper measures voltage on that wire.

Now the finding. The Hausa passenger walks in carrying the same weapon. Check-in scans him and correctly records what he’s carrying — that’s the 0.95–0.996 cosine similarity, the terminal’s paperwork is accurate, the system knows what this bag contains. But the corridor from the Hausa check-in desk never connects to the X-ray lane. He walks to the gate through a side hallway. The alarm wire reads under 10% of the voltage it reads for the English passenger. Not because the scanner is broken, and not because nobody logged the weapon — because the routing between the desk and the checkpoint was only ever built for English traffic.

The literal-versus-localized drift is the second half of the metaphor. Two passengers carry functionally equivalent contraband: one has a translated English-style weapon, the other a locally-made one. At check-in they look nearly the same (high cosine). Deeper into the terminal, staff route them to different hallways (layerwise drift). So even if you built one new corridor to the checkpoint for translated prompts, the locally-authored ones would still walk past it. That’s the practical sting: patching translated benchmarks does not patch the culturally native attack surface.

Key Concepts

• Refusal direction (difference-in-means probing): Inside a transformer, every prompt becomes a long list of numbers at each layer — call it a point in a high-dimensional space. If you take 200 harmful English prompts and 200 harmless ones and average each group’s point, the arrow from the harmless average to the harmful average is one particular direction in that space. Empirically, that single arrow is remarkably meaningful: push a prompt’s representation along it and the model becomes more likely to refuse; subtract it and refusals collapse, even for clearly harmful requests. It behaves like a scalar dial labeled “this is dangerous.” So the natural question becomes: when I feed the model a harmful Amharic prompt, how far does it move along that dial? Answer in this paper: barely. Under 10% of the English deflection.

• Semantic encoding vs. safety routing: These are separable, and the paper’s whole contribution rests on the separation. Encoding means the information is physically present in the activations — you could train a probe to read it out. Routing means that information actually influences the downstream computation that produces behavior. Analogy: a smoke detector’s battery is charged and the sensor detects smoke, but the wire to the siren was never soldered. Diagnostics say “smoke present.” The house stays quiet. Prior generation-based work could only observe the silence; this paper checks the sensor and finds it working, which tells you where to solder.

• Literal vs. localized harm: A literal prompt is the English harm question, translated. A localized prompt asks for something comparably dangerous but grounded in the actual context — say, a harmful traditional-remedy preparation, or a locally specific fraud script, or content targeting a locally salient ethnic fault line. The distinction matters for two reasons. Evaluation validity: a literal prompt can be so contextually alien that neither compliance nor refusal means much. And coverage: safety training built from translated data will have seen nothing resembling the localized attack surface. High cosine similarity between the two forms plus growing layerwise divergence is the paper’s evidence that they are semantically twinned but computationally handled apart.

Framework Shift

Before (mainstream approach):          After (this paper):

 harmful prompt (low-res lang)          harmful prompt (low-res lang)
        |                                       |
        v                                       v
   [   MODEL   ]  <-- black box            [ M O D E L ]
        |                                    |   |   |
        v                                    v   v   v
    generation                           h_1  h_k  h_L  (per layer)
        |                                    |   |   |
        v                                    +---+---+
   [ LLM judge ]                                 |
   refuse? comply?                               v
        |                              project onto r (EN refusal axis)
        v                                        |
   ASR = 71%                                     v
        |                            "meaning present (cos .99)
        v                              but < 10% on refusal axis"
  "it jailbreaks"                              |
  (why? unknown:                               v
   bad Twi? bad align?)              "encoded, not routed"
                                     (actionable diagnosis)

From attack-success rate to activation geometry, the core shift is that safety is measured as an internal signal path rather than an output label — which turns “it jailbreaks” into “the harm is understood but never reaches the refusal circuit.”

Expert Assessment

Problem choice: Real gap, well chosen. The low-resource jailbreak phenomenon has been known since 2023, and the field’s response has mostly been more benchmarks with more languages — a saturating direction. Asking *why it fails, with a mechanistic probe, is the right next move, and the literal/localized split is the kind of thing that only comes from a team with genuine native-language expertise rather than a translation API budget. It sits at the confluence of two live threads (multilingual safety, refusal-direction interpretability), and joining them is a natural but genuinely un-done step. The one thing I’d push back on: framing it as “strong evidence against a universal harm manifold” is a slightly grandiose reading of what is really a routing/coverage result — the abstract’s own hedge (“within the specific low-resource languages studied”) is doing a lot of work.

Method maturity: Clever reuse rather than new machinery. Difference-in-means refusal directions are off-the-shelf, and applying them cross-lingually is an obvious-in-hindsight move. That’s fine; the value is in the dataset and the framing. But the method is thinner than “latent geometric framework” implies. Two simpler controls I’d want and suspect are underweighted: (1) a high-resource non-English baseline — French, Chinese, Spanish. If French also retains only 40% of the English refusal signal, the story becomes “refusal directions are language-specific by construction,” which is a much weaker claim than “low-resource languages are unsafe.” Some deflation is expected purely because r was estimated on English activations, and the paper needs to show the low-resource numbers are anomalous, not just non-unity. (2) Perplexity or comprehension controls — the drift-with-depth result is exactly what you’d also expect if the model’s grasp of Twi degrades in deeper layers, so “encoding without routing” needs a positive comprehension check (e.g., a translation probe or topic probe that succeeds at the same layers where refusal projection fails). Without those, the headline conclusion is plausible but not nailed down.

Experimental integrity: The cosine numbers (0.95–0.996) are suspiciously high and deserve scrutiny. Hidden-state cosine similarity in transformer residual streams is notoriously inflated by a large shared component — the anisotropy problem. Two *unrelated prompts often sit at 0.9+ before centering. If those figures are raw cosines rather than mean-centered, they may show much less about semantic alignment than claimed, and “literal and localized are semantically aligned” would be partly an artifact. The <10% retention figure is more robust in direction, if not in exact magnitude, since it’s a relative comparison against the English condition. Four languages and (presumably) a handful of open-weight models is a modest footprint; the abstract’s careful scoping is appropriate but also limits how much the result generalizes. And a dataset of harmful localized prompts authored by native speakers is inherently hard to audit externally — I’d want inter-annotator agreement on harmfulness reported prominently, not in an appendix.

Writing quality: The dataset section is where the real contribution lives and, judging from the abstract’s proportions, where the least page space likely goes. LoDNA’s construction protocol — who authored the localized prompts, against what harm taxonomy, with what agreement, how the literal/localized pairing was validated as *matched in severity — is the load-bearing claim of the whole paper, and severity matching is essential: if localized prompts are simply milder, everything else follows trivially. Rewriting that section with full protocol detail and agreement statistics would elevate the paper more than any additional model. The “latent geometric framework” branding is oversold for what is a projection onto a difference-in-means vector; plainer language would build more trust.

Verdict: weak accept — the dataset and the encoded-but-not-routed framing are genuinely valuable contributions, but the causal claim needs a high-resource control language and a comprehension baseline before “encoded, not routed” is established rather than argued.

Takeaways

Things a practitioner can actually lift:

1. Use a known interpretability direction as a cross-domain ruler. The trick generalizes far past languages: estimate a behavioral direction (refusal, sycophancy, uncertainty, tool-use intent) on the distribution where you *know the behavior works, then measure retained projection on any distribution you’re worried about. This gives you a safety coverage metric that doesn’t need a judge model — valuable anywhere your judge is weaker than your model, which is most non-English and most specialized-domain settings.

2. Separate “does it know?” from “does it act?” in every eval you build. If a safeguard fails, run a probe to check whether the relevant information is present in activations. Present-but-unused points at alignment/routing fixes (targeted fine-tuning, activation steering on the deficient distribution). Absent points at pretraining data and tokenization. These consume very different budgets, and generation-only evals cannot tell you which you’re facing.

3. Pair literal with localized in every non-English eval set. This is a cheap, high-yield design pattern. If your translated set passes and your localized set fails, you’ve learned that your safety training memorized surface forms. If both fail equally, translation-based augmentation might actually help. Most teams only build the translated half and thereby cannot distinguish these.

4. Distrust raw cosine similarity in residual streams. Center your activations before comparing. If you’re reporting 0.99 similarities, first check what two random prompts score under the same measurement.

5. The negative-result-as-metric framing. “Retained fraction of the reference signal” is a nice reusable form for regression-testing alignment: pick a canonical direction once, then track retention as you ship new languages, domains, or fine-tunes.