Code2LoRA: Zero-Token Repository Context via Hypernetwork-Generated Adapters

Paper: 2606.06492 Authors: Liliana Hotsko, Yinxi Li, Yuntian Deng, Pengyu Nie Categories: cs.SE, cs.AI, cs.CL

The Gap

Code language models need repository-level context — imports, APIs, project conventions — to generate useful completions. Two existing solutions dominate: RAG-based context injection (retrieve relevant code snippets and stuff them into the prompt) and per-repository fine-tuning (train a LoRA adapter for each repo). RAG adds hundreds or thousands of tokens per query, eating inference budget. Per-repo fine-tuning requires training on every repository and breaks when code evolves — you retrain from scratch or accept stale adapters. At scale, neither approach is economical: thousands of repositories × thousands of commits = infeasible training cost.

The gap: we lack a method that (1) injects repository knowledge without token overhead, (2) generates adapters without per-repo training, and (3) handles evolving codebases incrementally.

Problem: Code LMs need repo context
   |
   v
Existing approaches:              Limitations:
RAG (retrieve + inject)    -->    +500-2000 tokens per query
Per-repo LoRA training     -->    Training cost × num_repos
                                  Brittle to code evolution
   |
   v
Gap: Need zero-token injection + zero per-repo training + evolution support
   |
   v
Method: Hypernetwork generates LoRA adapters from code embeddings
   |
   v
Evidence: 63.8% cross-repo EM (matches per-repo LoRA), +5.2 pp on evolution track
   |
   v
Conclusion: Hypernetworks amortize training cost, support evolution, eliminate token overhead

The Increment

One sentence: Before this paper, repository knowledge required either token-heavy retrieval or per-repository training; after, a single hypernetwork converts any repository snapshot or commit stream into a LoRA adapter with zero inference tokens.

Core Mechanism

Code2LoRA has two components: a code encoder and a hypernetwork. The code encoder (CodeBERT) converts repository code into embeddings. The hypernetwork is a neural network that takes these embeddings and outputs LoRA weight matrices — the adapter parameters that will modify a base code language model.

Two variants exist. Code2LoRA-Static processes a single repository snapshot: embed all files, aggregate embeddings (mean pooling), feed to hypernetwork, output one static LoRA adapter. Code2LoRA-Evo maintains a hidden state updated per commit: embed the diff, update a GRU hidden state, feed GRU state to hypernetwork, output an adapter that reflects the current codebase state. Each commit produces a new adapter without retraining the hypernetwork.

At inference, the generated adapter plugs into the base language model (CodeGen-350M or StarCoder2-3B). No repository code appears in the prompt. The adapter’s injected knowledge handles imports, APIs, conventions implicitly through modified attention and feedforward weights.

Code2LoRA-Static:                Code2LoRA-Evo:
                                  
Repo files                        Commit stream
   |                                 |
   v                                 v
CodeBERT encoder                  CodeBERT encoder (diff)
   |                                 |
   v                                 v
Aggregate (mean)                  GRU hidden state (updated)
   |                                 |
   v                                 v
Hypernetwork                      Hypernetwork
   |                                 |
   v                                 v
LoRA adapter W_A, W_B             LoRA adapter W_A, W_B (per commit)
   |                                 |
   +--------+------------------------+
            |
            v
   Base LM + adapter --> code completion
   (zero prompt tokens for repo context)

Think of the hypernetwork as a key duplicator. You bring your repository (the original key’s shape) to the machine. The machine scans it, then fabricates a custom adapter key (LoRA weights) that unlocks repository-specific knowledge inside the base model’s lock. The base model is unchanged; you just carry the right key.

For evolution, imagine the key duplicator has memory. Each time you bring a code diff (a modification to the key’s shape), the machine updates its internal blueprint (GRU state) and prints a new key reflecting all changes so far. You never re-scan the entire original key — just feed incremental modifications.

Key Concepts

• LoRA (Low-Rank Adaptation): Fine-tuning a billion-parameter model is expensive. LoRA freezes the original model and injects small trainable matrices $W_A$ and $W_B$ (rank $r \ll d$) into each layer: output = original_output + $W_B W_A x$. Training only $W_A$ and $W_B$ (a few million parameters) is cheap. For code models, a per-repository LoRA captures project-specific patterns. The problem: you need to train one LoRA per repository. If you have 10,000 repositories, you train 10,000 times.

• Hypernetwork: Instead of training parameters directly, train a network that generates parameters. A hypernetwork $H$ takes a description $z$ (e.g., repository embeddings) and outputs weights $W = H(z)$. Once $H$ is trained on many repositories, generating weights for a new repository requires one forward pass through $H$, not gradient descent. It’s parameter synthesis rather than parameter learning. In Code2LoRA, $H$ generates LoRA matrices: $W_A, W_B = H(\text\{repo\_embedding\})$.

• Assertion-Completion Task: The benchmark evaluates whether models can complete assertions in unit tests. Given a test function with the assertion line masked (e.g., assert result == ___), the model must predict the right-hand side. This task requires understanding function behavior, API contracts, and project conventions — exactly the knowledge that repository context should provide. It’s more realistic than next-token prediction on arbitrary code snippets.

Framework Shift

Before (RAG or per-repo training):        After (Code2LoRA):

Query time:                                Query time:
User prompt                                User prompt
   |                                          |
   v                                          v
Retrieve context (500-2000 tokens)         Base LM + adapter (0 tokens)
   |                                          |
   v                                          v
Base LM (prompt + context)                 Code completion
   |
   v
Code completion

Training time:                             Training time:
Per-repo gradient descent                  Train hypernetwork once
(repeat for each repo)                        |
                                              v
                                           Generate adapter per repo
                                           (one forward pass)

From retrieval-augmented generation to retrieval-free adaptation, the core shift is replacing runtime context with baked-in adapter weights.

Expert Assessment

Problem choice: Real gap. Repository context is a recognized bottleneck in code generation. RAG’s token overhead is painful in production, and per-repo fine-tuning doesn’t scale to large code hosts (GitHub has 100M+ repositories). The evolution angle is especially compelling — most code LM work assumes static codebases, but real development is incremental. The authors identified a problem that practitioners feel daily.

Method maturity: Hypernetworks are not new (introduced 2016), but applying them to LoRA generation for code is a clean fit. The GRU-based evolution mechanism is straightforward — arguably the simplest incremental update strategy. Could they have used a Transformer instead of GRU? Maybe, but GRU is cheap and the results justify it. The method feels like an obvious-in-retrospect synthesis of existing ideas rather than a deep algorithmic contribution, but sometimes that’s what moves the field forward.

Experimental integrity: The benchmark (RepoPeftBench) is purpose-built, which raises the usual concern: did they overfit to their own test set? They provide cross-repo and in-repo splits, and cross-repo results (63.8%) are only 2.4 pp below in-repo (66.2%), suggesting decent generalization. Baselines are fair: they compare against shared LoRA (one adapter for all repos) and per-repo LoRA (the upper bound). However, they don’t compare against strong RAG baselines with modern retrieval (e.g., BM25 + reranking). The RAG comparison is mentioned but not thoroughly benchmarked. The evolution track results (+5.2 pp over shared LoRA) are modest but meaningful. One red flag: the absolute numbers (60-66% exact match) are not spectacular. This suggests the task is hard or the models are small (CodeGen-350M, StarCoder2-3B). Larger base models might narrow the gap between methods.

Writing quality: The abstract and introduction are crisp. The method section is clear but could use a complexity analysis — what’s the memory overhead of storing one adapter per repository? How does hypernetwork inference time compare to per-repo LoRA training time? The evaluation section front-loads tables without enough narrative context. A paragraph walking through Table 2 (static track results) before presenting it would help. The related work section is thorough but reads like a laundry list. Cutting it by 30% and integrating key comparisons into the method section would tighten the paper.

Verdict: Weak accept — the problem is real, the method is practical, and the results are solid if unspectacular. The benchmark contribution (RepoPeftBench) adds value. However, the method’s novelty is incremental (hypernetworks applied to LoRA), and the evaluation could be more comprehensive (stronger RAG baselines, larger base models). It’s a good systems paper that makes code LMs more deployable, not a breakthrough in modeling.

Takeaways

For practitioners: If you operate code completion at scale (e.g., GitHub Copilot, Replit), this approach lets you support thousands of repositories without training thousands of adapters. Generate an adapter per repo on-demand, cache it, and swap at inference time. The zero-token overhead is a real cost saving — 1000 tokens per query × millions of queries = significant compute.

For researchers: The hypernetwork-as-parameter-generator pattern transfers to any domain where you need task-specific or user-specific adapters. Medical LMs (per-patient adapters), legal LMs (per-jurisdiction adapters), even personalized assistants (per-user adapters) could use this. The GRU-based incremental update is a template for handling any evolving context (user preferences over time, document versions, knowledge base updates).

Steal this: The assertion-completion task is a better benchmark than next-token prediction for evaluating repository-level understanding. If you’re building a code LM benchmark, consider goal-oriented tasks (write a test, fix a bug, implement a function spec) over perplexity on random code snippets.