Paper: 2604.20556 Authors: Yuhang Wu, Qinyuan Liu, Qiuyang Zhao, Qingwei Chong Categories: cs.AI, cs.CL
Problem
The evolutionary laws of hierarchical representations, task knowledge formation positions, and network robustness bottleneck mechanisms in various LLM architectures remain unclear.
LayerTracer Framework
An architecture-agnostic end-to-end analysis framework compatible with any LLM architecture (Transformer, GateDeltaNet, Mamba).
Method
- Extract hidden states layer-by-layer
- Map to vocabulary probability distributions
- Joint analysis of:
- Task particle localization
- Layer vulnerability quantification
Key Definitions
Task Particle
The key layer where target token probability first rises significantly - represents the model’s task execution starting point.
Vulnerable Layer
The layer with maximum Jensen-Shannon (JS) divergence between output distributions before and after mask perturbation - reflects sensitivity to disturbances.
Results
Across models of different parameter scales:
- Task particles mainly appear in deep layers regardless of parameter size
- Larger-parameter models exhibit stronger hierarchical robustness
- LayerTracer provides scientific basis for:
- Layer division
- Module ratio
- Gating switching in hybrid architectures
Takeaways
- Task execution starts later than expected (deep layers)
- Model size correlates with robustness
- Framework enables data-driven architecture design decisions
论文: 2604.20556 作者: 吴宇航、刘沁媛、赵秋阳、种庆伟 分类: cs.AI, cs.CL
问题
各种LLM架构中层级表示的进化规律、任务知识形成位置和网络鲁棒性瓶颈机制仍不明确。
LayerTracer框架
一个架构无关的端到端分析框架，兼容任何LLM架构（Transformer、GateDeltaNet、Mamba）。
方法
- 逐层提取隐藏状态
- 映射到词汇概率分布
- 联合分析：
- 任务粒子定位
- 层脆弱性量化
关键定义
任务粒子
目标token概率首次显著上升的关键层——代表模型任务执行的起点。
脆弱层
掩码扰动前后输出分布间Jensen-Shannon（JS）散度最大的层——反映对扰动的敏感性。
实验结果
跨不同参数规模的模型：
- 任务粒子主要出现在深层，无论参数大小
- 更大参数模型表现出更强的层级鲁棒性
- LayerTracer为以下提供科学依据：
- 层划分
- 模块比率
- 混合架构中的门控切换
要点总结
- 任务执行比预期开始得更晚（深层）
- 模型规模与鲁棒性相关
- 框架支持数据驱动的架构设计决策