SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

Paper: 2606.13647 Authors: Marek Šuppa, Andrej Ridzik, Daniel Hládek, Natália Kňažeková, Viktória Ondrejová Categories: cs.CL, cs.AI, cs.LG

The Gap

Existing work on text embeddings is dominated by English (MTEB) and a handful of high-resource languages. Multilingual benchmarks like BeIR and XMTEB include Slovak but only as a tiny fraction—often 1–3 datasets. Meanwhile, Slovak-specific NLU models (e.g., SlovakBERT, sloBERTa) were trained for classification/sequence labeling, not for general-purpose dense retrieval or sentence similarity. The community lacked: (1) a structured benchmark covering diverse Slovak embedding tasks, (2) an evaluation of how well existing multilingual and Slovak models perform on those tasks, and (3) efficient, locally-deployable Slovak embedding models.

This paper fills all three gaps. It constructs SkMTEB (31 datasets, 7 task types), evaluates 31 models, and then develops e5-sk-small (45M) and e5-sk-large (365M) by trimming the vocabulary of Multilingual E5 and fine-tuning on SkMTEB training data.

+---

![Concept animation](/arxiv-visuals/skmteb-slovak-massive-text-embedding-benchmark/ConceptScene.gif)

---------------------------------+      +-----------------------------------+
| Problem:                           | ---> | Assumption:                       |
| no comprehensive Slovak embedding  |      | benchmark + careful model         |
| benchmark, no efficient local      |      | adaptation can close the gap      |
| Slovak embedding model             |      |                                   |
+------------------------------------+      +-----------------------------------+
        |                                                |
        v                                                v
+------------------------------------+      +-----------------------------------+
| Method:                            |      | Evidence:                         |
| build SkMTEB (31 datasets)         |      | 31 models eval shows:             |
| develop e5-sk via                  |      | (a) multilingual instruction      |
| vocabulary trimming + fine-tuning  |      |     models win on SK tasks        |
| on Multilingual E5                 |      | (b) e5-sk models match            |
|                                    |      |     proprietary API performance   |
+------------------------------------+      +-----------------------------------+
        |                                                |
        +------------------------+-----------------------+
                                 |
                                 v
              +-------------------------------------+
              | Conclusion:                         |
              | SkMTEB + e5-sk models = ready-to-  |
              | use infrastructure for Slovak NLP; |
              | recipe replicable for other low-   |
              | resource languages                 |
              +-------------------------------------+

The Increment

One sentence: Before this paper, Slovak had no benchmark for embedding tasks and no efficient open-source embedding model; after this paper, it has both a 31-dataset benchmark and 45M/365M models that rival proprietary APIs in performance while being deployable on a single CPU.

Core Mechanism

The authors start with Multilingual E5 (available in base and large sizes), which already produces strong multilingual embeddings but carries a large vocabulary of 250k+ tokens optimized for 93 languages. For Slovak, many of those tokens are rarely or never used, wasting memory and compute.

They perform vocabulary trimming: given a Slovak corpus (e.g., a subset of SkMTEB training data), they compute token frequencies using the E5 tokenizer, then keep only the top K tokens that appear at least once, plus a fallback byte-level coverage set. This produces a new vocabulary of about 20k–30k tokens for Slovak. The corresponding embedding matrix rows (and all transformer head weights) are extracted from the original model to form the new model.

Then they fine-tune this trimmed model on multiple embedding tasks from SkMTEB (STS, classification, clustering, etc.) using a multi-task contrastive loss. The result is e5-sk-small (45M parameters, from the original 118M base) and e5-sk-large (365M, from the original 960M large), with size reductions of up to 62%.

[Multilingual E5 (large vocabulary, 93 languages)]
        |
        v
[Tokenize Slovak corpus, count token frequencies]
        |
        v
[Select top-K tokens + byte fallback --> new vocab ~20-30k]
        |
        v
[Extract corresponding embedding matrix rows + transformer weights]
        |
        v
[New model: same architecture, smaller vocab & embedding layer]
        |
        v
[Multi-task fine-tuning on SkMTEB training datasets]
        |
        v
[e5-sk-small (45M) / e5-sk-large (365M)]

Think of it like translating a massive encyclopedia set into a specialized pocket guide. The original Multilingual E5 is a 93‑volume encyclopedia covering every language. Most volumes are irrelevant to a Slovak reader. Vocabulary trimming is like pulling out only the Slovak-relevant pages and rebinding them into a thin book. The two volumes still contain all the high‑level chapter structures (the transformer layers) and illustrations (the attention weights), but the index (vocabulary) is now tailored to Slovak. Fine‑tuning is like adding a new preface and footnotes that connect the general knowledge to Slovak‑specific examples—making the pocket guide even better for local use. The pocket guide is much lighter (45M vs 118M) yet covers the same semantic territory because the core encyclopedic knowledge in the transformer layers was already well‑trained; you just needed to fix the vocabulary mismatch and adapt to local tasks.

Key Concepts

• MTEB (Massive Text Embedding Benchmark): A standardised suite of tasks (sentence similarity, classification, clustering, reranking, retrieval, STS, etc.) used to evaluate how well an embedding model captures semantic meaning. Before MTEB, each paper used its own tasks and data, making comparison impossible. MTEB provides a common yardstick. SkMTEB brings this to Slovak: same task categories, but all data in Slovak (translated, curated, or native). Intuitively, if you want to know whether a Slovak embedding model can tell that “auto” (car) and “vozidlo” (vehicle) are similar while “auto” and “jablko” (apple) are not, MTEB gives you a score on that ability.

• Vocabulary Trimming: Multilingual models allocate a giant vocabulary (e.g., 250k tokens) to cover many languages. For a single language like Slovak, only ~10-15% of those tokens ever appear. Vocabulary trimming discards the unused tokens and their associated weights. This shrinks the model significantly and speeds up inference, because every token embedding lookup now returns from a much smaller matrix. The key insight is that the transformer’s internal representations (the “thinking” layers) are largely language‑agnostic; most of the weight overhead is in the embedding layer. Trimming removes that overhead with minimal damage to quality, especially if you then fine‑tune on target tasks.

• Instruction‑tuned Embedding Models: Models like E5 are trained not just to produce generic embeddings but to follow a short instruction prepended to the query (e.g., “Represent the sentence for classification:”). This instruction conditions the embedding on the intended task, dramatically improving performance compared to fixed embeddings. The authors find that instruction‑tuned multilingual models outperform all Slovak‑specific models that lack instruction conditioning. For practitioners, this means you should always use an instruction‑style model when possible; in low‑resource settings, you can adopt a multilingual instruction model and trim it down.

Framework Shift

Before (mainstream approach):                After (this paper):

+-----------------------------------+        +----------------------------------+
| Use any off-the-shelf multilingual |        | Build a benchmark first (SkMTEB) |
| embedding (E5, LaBSE, etc.)       |        | to know what you need.            |
| or a Slovak NLU model (slovakbert)|        | Then take a multilingual model,   |
| and hope it works for retrieval.  |        | trim its vocab to Slovak,         |
| No benchmark to validate.         |        | fine-tune on the benchmark tasks. |
| No efficient local model exists.  |        | Result: small, efficient,         |
|                                   |        | locally deployable model with     |
|                                   |        | near‑API performance.             |
+-----------------------------------+        +----------------------------------+

One sentence: From “use whatever multilingual or task‑specific model you can find and guess its quality” to “systematically benchmark then prune‑and‑adapt a strong multilingual base into a compact country‑specific model,” the core shift is building infrastructure (benchmark + trained model) for a low‑resource language from scratch, then showing the recipe works.

Expert Assessment

Problem choice: Genuine gap. Slovak is a low‑resource West Slavic language with ~5 million speakers. The lack of a benchmark and efficient embedding models directly blocks RAG and semantic search applications in Slovak language technology. This sits at the intersection of multilingual NLP and low‑resource language engineering, a well‑motivated and timely area.

Method maturity: Vocabulary trimming + fine‑tuning is not conceptually novel—similar ideas appear in domain‑adaptation for language models (e.g., reducing vocabulary for code). However, applying it to embedding models and providing full benchmark infrastructure is a solid engineering contribution. The authors could have explored distillation or adapter‑based methods, but trimming is simpler and works well. No red flags.

Experimental integrity: The evaluation covers 31 models across 31 datasets, which is thorough. Baselines include multilingual giants (E5, LaBSE, BGE) and Slovak‑specific NLU models. The comparison is fair—they even test instruction‑tuned vs non‑instruction variants. One minor concern: the fine‑tuning uses SkMTEB training data, which overlaps with some test datasets (they note this and use separate splits). The paper would benefit from an ablation study isolating the effect of trimming vs fine‑tuning, but the evidence is still compelling.

Writing quality: The paper is clear but dense; the abstract and introduction do a good job of setting up the gap. The methods section could be more precise about token selection criteria (top‑K frequency vs byte fallback ratio). If I had to pick one section to rewrite, it would be the “Model Adaptation” subsection: it currently reads like a recipe; adding a small ablation table would elevate it from “we did this” to “this is why it works.”

Verdict: Strong accept — solid infrastructure contribution for Slovak NLP, replicable methodology, open‑source everything.

Takeaways

• For any low‑resource language, the pipeline is clear: (1) build a small MTEB‑style benchmark using translated or curated data, (2) pick a strong instruction‑tuned multilingual embedding model, (3) trim its vocabulary to your language using a corpus, (4) fine‑tune on your benchmark tasks. The paper provides code and datasets that can be adapted.
• Vocabulary trimming alone gives ~60% size reduction with minimal quality loss (~1–2% on most tasks). This is a cheap and effective trick for model compression that any team working with a specific language can steal.
• Instruction‑tuned models dominate on Slovak embedding tasks, even compared to Slovak‑specific NLU models. Practitioners should abandon pure NLU‑style pretrained models for retrieval/similarity and use instruction‑conditioned multilingual ones instead.