论文标题：Attention Is All You Need

作者：Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin（Google Brain / Google Research）arXiv

提交时间：2017-06-12

原文 PDF：https://arxiv.org/pdf/1706.03762.pdf

网盘下载地址：

链接：https://pan.quark.cn/s/9f868c118e41?pwd=y9Sj

提取码：y9Sj

摘要：

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

摘要翻译：

主流的序列转换模型基于带有编码器、解码器结构的复杂循环神经网络或卷积神经网络；效果最优的模型还会借助注意力机制连接编码器与解码器。本文提出一种简洁的全新网络架构 Transformer，该架构完全依托注意力机制，彻底舍弃循环结构与卷积操作。在两项机器翻译任务上的实验表明：该模型翻译质量更优、并行能力更强、训练耗时大幅缩短。在 WMT 2014 英德翻译任务上取得 28.4 BLEU，相比包含集成模型在内的此前最优结果高出 2 BLEU 以上；在 WMT 2014 英法翻译任务上，使用 8 块 GPU 训练 3.5 天后，单模型取得 41.8 BLEU 的新 SOTA 结果，训练成本远低于文献中其他最优模型。同时，我们将 Transformer 应用于英语成分句法分析（大数据、小数据场景均做测试），证明它可以很好地泛化到其他任务。

论文要点：

  1. 核心创新：抛弃 RNN / CNN，整个序列模型只依靠自注意力（Self-Attention），提出 Transformer 架构，支持高度并行训练

  2. 缩放点积注意力公式

\(Attention(Q,K,V) = softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V\)

\(\sqrt{d_k}\) 缩放是为了防止维度高时点积值过大，softmax 梯度变稀疏arXiv

3. 多头注意力 Multi-Head Attention：多组独立 QKV 投影，捕捉不同类型依赖

4. 结构：Encoder（双向自注意力）+ Decoder（掩码自注意力 + 交叉注意力）+ 位置编码 Positional Encoding + FFN + LayerNorm + Residual