transformer(attention is all your need)

1.We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

2.Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.

1 Positional Encoding

in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence. There are many choices of positional encodings, learned and fixed [9]. In this work, we use sine and cosine functions of different frequencies:

$$PE_{(pos,2i)}=sin(pos/10000^{2i/d_{model}})\\ PE_{(pos,2i+1)}=cos(pos/10000^{2i/d_{model}})$$

详细可参考 https://wmathor.com/index.php/archives/1453/

2 Attention

其中不同颜色表示不同head，颜色深浅表示词的关联程度。

不同head表示不同应用场景 ，单一head表示某个场景下，各个字之间的关联程度

1 Scaled Dot-Product Attention

$$Attention(Q,K,V)=softmax(\frac{QK^T}{\sqrt{d_{k}}})V$$

$d_{k}$ ： keys of dimension

为什么scale？We suspect that for large values of dk, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients.

Mask

可以分为两类：Attention Mask和Padding Mask，接下来具体讲解。

1.Attention Mask

ensures that the predictions for position i can depend only on the known outputs at positions less than i.

sotfmax前要mask，上三角mask掉

2.Padding Mask

Padding位置上的信息是无效的，所以需要丢弃。

过程如下图示：

2 Multi-Head Attention

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.

$$MultiHead(Q,K,V)=Concat(head1,...,head_n)W^O \\ where \ head_{i}=Attetion(QW_{i}^{Q},kW_{i}^{K},VW_{i}^{V})$$

3 Applications of Attention in our Model

1.encoder-decoder attention layers

结构：queries come from the previous decoder layer,and the memory keys and values come from the output of the encoder.

目的：This allows every position in the decoder to attend over all positions in the input sequence.

2.encoder contains self-attention layers

结构：keys, values and queries come from the same place

目的：Each position in the encoder can attend to all positions in the previous layer of the encoder.

3.self-attention layers in the decoder

结构：keys, values and queries come from the same place

目的：allow each position in the decoder to attend to all positions in the decoder up to and including that position

3 Encoder and Decoder Stacks

1 encoder

1).Input Embedding与Positional Encoding

$$X = \text{Input Embedding}+ \text{Positional Encoding}\\$$

2). multi-head attention

$$Q = \text{Linear}_q(X) = XW_{Q}\\ K = \text{Linear}_k(X) = XW_{K}\\ V = \text{Linear}_v(X) = XW_{V}\\ X_{attention} = \text{Attention}(Q,K,V)$$

3). 残差连接与 Layer Normalization

$$X_{attention} = X + X_{attention}\\ X_{attention} = \text{LayerNorm}(X_{attention})$$

4). FeedForward

$$X_{hidden} = \text{Linear}(\text{ReLU}(\text{Linear}(X_{attention})))$$

5). 残差连接与 Layer Normalization

$$X_{hidden} = X_{attention} + X_{hidden}\\ X_{hidden} = \text{LayerNorm}(X_{hidden})$$

其中$ X_{hidden} \in \mathbb{R}^{batch_size \ \ seq_len \ \ embed_dim} $

2 decoder

我们先从 HighLevel 的角度观察一下 Decoder 结构，从下到上依次是：

• Masked Multi-Head Self-Attention
• Multi-Head Encoder-Decoder Attention
• FeedForward Network

4 常见问题

1 并行化

训练encoder，decoder都并行，测试encoder并行，decoder不是并行

https://zhuanlan.zhihu.com/p/368592551

2 self-attention和普通attention的区别

取决于query和key是否在一个地方

3 Why Self-Attention

Motivating our use of self-attention we consider three desiderata.

1.One is the total computational complexity per layer.

2.Another is the amount of computation that can be parallelized, as measured by the minimum number of sequential operations required.

3.The third is the path length between long-range dependencies in the network

参考

https://arxiv.org/abs/1706.03762

大佬详解： https://jalammar.github.io/illustrated-transformer/