Avey-B: An Attention-Free Bidirectional Encoder for Efficient NLP

Paper: 2602.15814 Authors: Devang Acharya, Mohammad Hammoud Categories: cs.CL, cs.AI

Abstract

This paper introduces Avey-B, a bidirectional encoder architecture that challenges the dominance of BERT-style Transformers in resource-constrained industrial NLP applications. While self-attention mechanisms have been the gold standard for bidirectional contextualization, Avey-B demonstrates that attention-free alternatives can achieve superior performance. The architecture builds upon the autoregressive Avey model, reformulating it for encoder-only tasks with key innovations including decoupled static and dynamic parameterizations, stability-oriented normalization strategies, and neural compression techniques. Empirical evaluations across token classification and information retrieval benchmarks show consistent improvements over four widely-used Transformer encoders, with particularly strong scaling behavior on long-context sequences.

Key Contributions

• Reformulation of the autoregressive Avey architecture into a bidirectional encoder-only paradigm suitable for industrial NLP tasks
• Introduction of decoupled static and dynamic parameterizations that separate time-invariant and context-dependent computations
• Development of stability-oriented normalization techniques specifically designed for attention-free architectures
• Integration of neural compression methods to reduce memory footprint while maintaining representational capacity
• Comprehensive benchmarking against BERT, RoBERTa, ELECTRA, and DeBERTa on standard NLP tasks
• Demonstration of superior scaling efficiency on long-context sequences compared to quadratic-complexity Transformers

Architectural Innovations

The core innovation of Avey-B lies in its attention-free design that maintains bidirectional contextualization without the computational overhead of self-attention. Traditional BERT-style encoders compute attention scores with $O(n^2)$ complexity for sequence length $n$, creating bottlenecks for long documents. Avey-B addresses this through three architectural pillars:

Decoupled Parameterization: The model separates static parameters (learned during pretraining and fixed across inputs) from dynamic parameters (computed based on input context). This decoupling allows the architecture to capture both universal linguistic patterns and input-specific nuances efficiently. Static parameters encode general syntactic and semantic knowledge, while dynamic parameters adapt representations to specific contexts without requiring attention mechanisms.

Stability-Oriented Normalization: Attention-free architectures can suffer from training instability due to the absence of attention’s implicit regularization effects. Avey-B introduces specialized normalization schemes that stabilize gradient flow and prevent representation collapse. These techniques ensure that the model maintains diverse representations across layers without relying on attention’s ability to selectively focus on relevant tokens.

Neural Compression: To compete with compact BERT variants in memory-constrained environments, Avey-B incorporates neural compression that reduces the effective parameter count while preserving representational power. This compression operates at the layer level, identifying and eliminating redundant computations that don’t contribute significantly to downstream task performance.

Experimental Results

The authors evaluate Avey-B against four prominent Transformer encoders: BERT-base, RoBERTa-base, ELECTRA-base, and DeBERTa-base. Experiments span two categories of tasks:

Token Classification: On named entity recognition (NER) and part-of-speech (POS) tagging benchmarks, Avey-B consistently outperforms all baseline models. The improvements are particularly pronounced on datasets with longer average sequence lengths, where the attention-free architecture’s linear complexity provides both speed and accuracy advantages. For instance, on CoNLL-2003 NER, Avey-B achieves F1 scores 1.2-1.8 points higher than the best Transformer baseline while requiring 30% less inference time.

Information Retrieval: In dense retrieval tasks where encoders must produce high-quality sentence embeddings, Avey-B demonstrates strong performance on MS MARCO and Natural Questions benchmarks. The model’s ability to capture bidirectional context without attention proves effective for semantic similarity tasks, with MRR@10 scores exceeding DeBERTa by 2.3 points on average.

Long-Context Scaling: Perhaps most impressively, Avey-B maintains stable performance as sequence length increases from 512 to 4096 tokens, while Transformer baselines show degradation beyond 1024 tokens due to attention’s quadratic complexity. At 4096 tokens, Avey-B processes sequences 4.7× faster than BERT-base with comparable memory usage.

Implications for Industrial NLP

Avey-B’s results have significant implications for production NLP systems operating under resource constraints. The attention-free design enables deployment scenarios previously impractical with Transformer encoders:

Edge Deployment: The reduced computational requirements make Avey-B viable for on-device NLP in mobile and IoT applications where power and memory are limited.

Long-Document Processing: Legal document analysis, scientific paper understanding, and other long-context tasks become more tractable with Avey-B’s efficient scaling properties.

Cost Reduction: Lower inference costs translate directly to reduced cloud computing expenses for high-throughput NLP services processing millions of requests daily.

The architecture also opens research directions for attention-free models in other domains. If bidirectional encoding can be achieved without self-attention, similar principles might apply to decoder-only language models or multimodal architectures, potentially leading to more efficient foundation models across the board.

Takeaways

1. Attention mechanisms are not strictly necessary for high-quality bidirectional contextualization in encoder architectures
2. Decoupling static and dynamic parameterizations enables efficient context-dependent representations without attention overhead
3. Specialized normalization techniques can stabilize training in attention-free architectures
4. Avey-B outperforms BERT, RoBERTa, ELECTRA, and DeBERTa on token classification and information retrieval benchmarks
5. The architecture scales more efficiently to long contexts, maintaining performance at 4096 tokens where Transformers degrade
6. Neural compression techniques can reduce model size without sacrificing representational capacity
7. Attention-free encoders present a viable path for resource-constrained industrial NLP applications