English
Overview
BERT (Bidirectional Encoder Representations from Transformers) revolutionized NLP by introducing deep bidirectional pretraining. Unlike previous models that read text left-to-right or right-to-left, BERT reads the entire sequence at once, enabling richer understanding.
Key Contributions
- Bidirectional Context: Attends to both left and right context simultaneously
- Masked Language Modeling (MLM): Randomly masks tokens and predicts them
- Next Sentence Prediction (NSP): Learns sentence relationships
- Transfer Learning: Pretrain once, fine-tune for many tasks
Architecture
- 12-layer (BERT-Base) or 24-layer (BERT-Large) Transformer encoder
- 110M to 340M parameters
- Trained on BooksCorpus and Wikipedia
Impact
BERT achieved state-of-the-art on 11 NLP tasks：
- Question answering (SQuAD)
- Natural language inference (MNLI)
- Named entity recognition
- Sentiment analysis
Spawned many variants： RoBERTa, ALBERT, DistilBERT, etc.
Paper Link
.04805中文
概述
BERT（来自Transformer的双向编码器表示）通过引入深度双向预训练彻底改变了NLP。与之前从左到右或从右到左读取文本的模型不同，BERT一次读取整个序列，实现更丰富的理解。
主要贡献
- 双向上下文：同时关注左右上下文
- 掩码语言建模（MLM）：随机掩盖词元并预测
- 下一句预测（NSP）：学习句子关系
- 迁移学习：预训练一次，微调用于多种任务
架构
- 12层（BERT-Base）或24层（BERT-Large）Transformer编码器
- 1.1亿到3.4亿参数
- 在BooksCorpus和维基百科上训练
影响
BERT在11个NLP任务上达到最先进水平：
- 问答（SQuAD）
- 自然语言推理（MNLI）
- 命名实体识别
- 情感分析
衍生出许多变体：RoBERTa、ALBERT、DistilBERT等。