Runtime-Reconfigurable Bitwise Systolic Arrays for Multi-Precision Neural Network Acceleration

Paper: 2602.23334 Authors: Yuhao Liu, Salim Ullah, Akash Kumar Categories: cs.AR, cs.AI

Abstract

This paper addresses a critical limitation in neural network accelerators: the inability to dynamically reconfigure computational precision at runtime for mixed-precision quantized neural networks (QNNs). While quantization reduces hardware resource consumption, uniform low precision causes significant accuracy degradation. The authors propose a runtime-reconfigurable multi-precision multi-channel bitwise systolic array architecture that enables different layers to operate at different precisions without hardware redesign. Implemented on the Ultra96 FPGA platform, the design achieves 1.32× to 3.57× speedup for mixed-precision model inference while supporting higher clock frequencies (250MHz) due to reduced critical path delay.

Key Contributions

• A bitwise systolic array architecture that supports runtime reconfiguration of multiplication precision for QNN accelerators
• Multi-channel processing capability enabling parallel computation across different precision requirements
• Hardware implementation demonstrating practical feasibility on resource-constrained FPGA platforms
• Significant performance improvements (up to 3.57× speedup) while maintaining flexibility for mixed-precision models
• Reduced critical path delay enabling higher operating frequencies compared to conventional designs

Architecture Design and Methodology

The core innovation lies in the bitwise decomposition of multiplication operations within the systolic array structure. Traditional systolic arrays perform fixed-precision matrix multiplications, making them inflexible for mixed-precision workloads. The proposed architecture breaks down multiplications into bit-level operations that can be dynamically composed based on the required precision.

The systolic array processes data in a pipelined fashion, where each processing element (PE) performs partial product accumulation at the bit level. By controlling which bit-level operations are activated, the system can effectively implement $2$-bit, $4$-bit, $8$-bit, or other precision levels without structural changes. This is achieved through:

• Configurable bit-serial multiplication units within each PE
• Dynamic routing logic that adapts data flow based on current precision requirements
• Multi-channel support allowing different channels to operate at different precisions simultaneously

The multi-channel capability is particularly important for modern neural networks where different feature maps may benefit from different quantization levels. The architecture can process multiple channels in parallel, each configured independently for optimal precision-accuracy tradeoffs.

Experimental Results and Performance Analysis

The implementation on the Ultra96 FPGA platform demonstrates the practical viability of the approach. The authors evaluated their design across various mixed-precision QNN models and compared against baseline fixed-precision implementations.

Performance gains range from 1.3185× to 3.5671× depending on the specific precision configuration and model architecture. The speedup is attributed to several factors:

• Efficient utilization of hardware resources by avoiding over-provisioning for worst-case precision
• Reduced data movement through optimized bit-level processing
• Higher clock frequency capability (250MHz) due to shorter critical paths in the bitwise design

The critical path analysis reveals that bitwise operations introduce less combinational delay compared to full-width multipliers, enabling higher frequency operation. This is counterintuitive but results from the fact that bit-level operations can be more deeply pipelined and have simpler logic structures.

Energy efficiency also improves as the system only activates the necessary computational resources for each layer’s precision requirement, avoiding wasted computation on unused high-order bits in low-precision layers.

Implications for Edge AI and Future Directions

This work has significant implications for deploying neural networks on edge devices where both computational efficiency and model accuracy are critical. The runtime reconfigurability enables:

• Adaptive precision based on input characteristics or battery levels
• Per-layer optimization without requiring multiple specialized hardware units
• Easier deployment of neural architecture search (NAS) results that often produce mixed-precision models

The bitwise systolic array approach opens several research directions. Future work could explore dynamic precision adjustment based on runtime conditions, integration with emerging quantization-aware training methods, and extension to other operations beyond multiplication such as activation functions and normalization layers.

The demonstrated 250MHz operating frequency on FPGA suggests that ASIC implementations could achieve even higher performance, making this architecture particularly attractive for next-generation edge AI accelerators where flexibility and efficiency must coexist.

Takeaways

1. Runtime-reconfigurable precision is essential for efficiently deploying mixed-precision QNNs on hardware accelerators
2. Bitwise systolic array decomposition enables flexible precision without sacrificing performance, achieving up to 3.57× speedup
3. Multi-channel support allows different feature maps to operate at optimal precision levels simultaneously
4. Reduced critical path delay in bitwise designs enables higher clock frequencies (250MHz) compared to conventional approaches
5. The architecture bridges the gap between fixed-precision accelerators and the growing need for adaptive, mixed-precision neural network inference