論文解讀 - An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

目前我認為在分類領域上，最有研究價值的是 ResNet、Transformer，一個是從 Papers with code 的 benchmark 判斷，另外一個是對以往 DenseNet、Inception 等網路的實驗，深度學習從 2012 年的 AlexNet 開始，然后是 VGG，網路是往更深的方向發展，直到遇到一個問題，網路越深，模型性能開始下降，在排除了梯度消失/爆炸以及過擬合問題之后，發現是網路的優化困難問題，這時候提出的 Residual Learning，也就是 ResNet 解決了該問題，此后的 DenseNet 等網路也都包含 Skip connection，從設計上來看并未脫離這個框架范疇，再者 BiT Transfer 論文也證明 ResNet 在 ImageNet 上可達到 88% 左右的準確率，與最高的基于 Transformer 架構的 90% 也相差無幾，

Transformer 目前在 NLP 占據主流地位，CNN 目前在 CV 占據主流地位，那么我們就會思考 Transformer 是否可以遷移應用到 CV 上，這里提一下 Transformer 的兩個點，self-attention、序列化處理，其中 attention 機制在 CNN 里有引入，但是效果一般般，而 Transformer 更加徹底，直接應用到影像上，

Self-attention-based architectures, in particular Transformers, have become the model of choice in natural language processing(NLP).

Transformer 架構目前在 NLP 領域占據統治地位，那么這個架構是否可以遷移到 CV 領域呢，在 CNN 架構中，我們會引入注意力模塊，然而注意力模塊在 Transformer 上很自然(它的本質是一個序列處理，注意力只是其中一個重要模塊，特點是 global self-attention)，所以是否可以把 Transformer 應用在影像上呢，這個思考很有價值，

We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks(ImageNet, CIFAR-100, VTAB). Vision Transformer(ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

解讀: 這里的出發點呢，是以前的 CNNs 網路里有添加注意力機制模塊，這里更加徹底，把整個網路用 Transformer 來替代，這里呢，對于 Vision Transformer 來說 ImageNet-1k 也只是中等規模的資料集，所以歸根結底，還是要大資料集來做預訓練，以提高泛化性，但是這里也要注意不同架構的網路，指 CNNs 和 Transformer，兩者之間的性能可能隨資料集規模的變化趨勢也會不一樣？下面的一句話，算是可以回答這個問題吧，我的理解是 Transformer 的上限更高吧，

With the models and datasets growing, there is still no sign of saturating performance.

Inspired by the Transformer scaling successes in NLP, we experiment with applying a standard Transformer directly to images, with the fewest possible modifications. To do so, we split an image into patches and provide the sequence of linear embeddings of these pathces as an input to a Transformer. Image patches are treated the same way as tokens(words) in an NLP application. We train the model on image classification in supervised fashion.

When trained on mid-sized datasets such as ImageNet without strong regularization, these models yield modest accuracies of a few percentage points below ResNets of comparable size. This seemingly discouraging outcome may be expected: Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amout of data.

However, the picture changes if the models are trained on larger datasets(14M-300M images). We find that large scale training trumps inductive bias. Our Vision Transformer(ViT) attains excellent results when pre-trained at sufficient scale and transferred to tasks with fewer datapoints. When pre-trained on the pubilc ImageNet-21k dataset or the in-house JFT-300M dataset, ViT approaches or beats state of the art on multiple image recognition benchmarks.

解讀: 對于 CNNs 來說，我認為只需要研究到 ResNet 就夠了，這個是因為看了 papers with code 的 benckmark 以及 BiT transfer 這篇論文之后的思考，Transformer 對于資料的要求更高，因為缺失 inductive biases，對比 ViT 和 BiT，其實兩者的準確率是差不多的，所以貌似也看不太出來 Transformer 比 CNNs 更有優勢吧，

When feeding images of higher resolution, we keep the patch size the same, which results in a larger effective sequence length. The Vision Transformer can handle arbitrary sequence lengths(up to memory constraints), however, the pre-trained position embeddings may no longer be meaningful. We therefore perform 2D interpolation of the pre-trained position embeddings, according to their location in the original image. Note that this resolution adjustment and patch extraction are the only points at which an inductive bias about the 2D structure of the images is manually injected into the Vision Transformer.

Navie application of self-attention to images would require that each pixel attends to every other pixel. With quadratic cost in the number of pixels, this does not scale to realistic input sizes.

Note that the Transformer’s sequence length is inversely proportional to the square of the patch size, thus models with smaller patch size are computationally more expensive.

解讀: 從上圖來看，ViT 的準確率優勢并不是特別大，但是訓練時間成本上確實很有優勢，達到相同精度對比情況下，

This result reinforces the intuition that the convolutional inductive bias is useful for smaller datasets, but for larger ones, learning the relevant patterns directly from data is sufficient, even beneficial.

解讀: 從上圖可以看出來，對于在 ImageNet-1k 上預訓練 ViT，其結果比 BiT 差，對于在 ImageNet-21k 上預訓練 ViT，其結果和 BiT 相當，對于在 JFT-300M 上預訓練，結果比 BiT 略微好一些，我的想法，目前來看，分類上我覺得用 CNNs 就行，畢竟模型訓練出來之后，我們還需要經過壓縮等手段，然后再部署上線，卷積網路具有歸納偏置，這個對于小資料集來說是有用的，但是對于更大型的資料集，我們可以直接學習其中的相關模式，這個本質上有點類似于上帝視角了，只要我們囊括了該場景下的所有情況，那么模型的效果就會很好，

Transformers show impressive performance on NLP tasks. However, much of their success stems not only from their excellent scalability but also from large scale self-supervised pre-training.

Transformers are networks that operate on sequences of data, for example, a set of words. These sets of works are first tokenized and then fed into the transformer. Transformers add Attention (a quadratic operation – calculates pairwise inner product between each pair of the tokenized words. As the number of words increases, so does the number of operations).

The authors of ViT solve this problem by using global attention, but not on the entire image, rather on multiple image patches. So first a large image is separated into multiple small patches (for example 16 x 16 pixels).

解讀: ViT 使用全域注意力機制，不同點在于把原始影像切分成多個 patches

However, if entire image data is fed into a model, rather than just the parts that the filters can extract (or it considers important), the chances of the model performing better are higher. This is exactly what is happening inside a Visual Transformer.

解讀: CNN 存在 inductive biases, Transformer 存在 global self-attention, 對于資料量越大的情況，資料的 relevant patterns 越全，這時候 Transformer 更有優勢，也可以說明它的全域掌控能力更強，

This example implements the Vision Transformer (ViT) model for image classification, and demonstrates it on the CIFAR-100 dataset. The ViT model applies the Transformer architecture with self-attention to sequences of image patches, without using convolution layers.

The PatchEncoder layer will linearly transform a patch by projecting it into a vector of size projection_dim. In addition, it adds a learnable position embedding to the projected vector.

class PatchEncoder(layers.Layer):
    def __init__(self, num_patches, projection_dim):
        super(PatchEncoder, self).__init__()
        self.num_patches = num_patches
        self.projection = layers.Dense(units=projection_dim)
        self.position_embedding = layers.Embedding(
            input_dim=num_patches, output_dim=projection_dim
        )

    def call(self, patch):
        positions = tf.range(start=0, limit=self.num_patches, delta=1)
        encoded = self.projection(patch) + self.position_embedding(positions)
        return encoded

The ViT model consists of multiple Transformer blocks, which use the layers.MultiHeadAttention layer as a self-attention mechanism applied to the sequence of patches. The Transformer blocks produce a [batch_size, num_patches, projection_dim] tensor, which is processed via an classifier head with softmax to produce the final class probabilities output.

Unlike the technique described in the paper, which prepends a learnable embedding to the sequence of encoded patches to serve as the image representation, all the outputs of the final Transformer block are reshaped with layers.Flatten() and used as the image representation input to the classifier head. Note that the layers.GlobalAveragePooling1D layer could also be used instead to aggregate the outputs of the Transformer block, especially when the number of patches and the projection dimensions are large.

def create_vit_classifier():
    inputs = layers.Input(shape=input_shape)
    # Augment data.
    augmented = data_augmentation(inputs)
    # Create patches.
    patches = Patches(patch_size)(augmented)
    # Encode patches.
    encoded_patches = PatchEncoder(num_patches, projection_dim)(patches)

    # Create multiple layers of the Transformer block.
    for _ in range(transformer_layers):
        # Layer normalization 1.
        x1 = layers.LayerNormalization(epsilon=1e-6)(encoded_patches)
        # Create a multi-head attention layer.
        attention_output = layers.MultiHeadAttention(
            num_heads=num_heads, key_dim=projection_dim, dropout=0.1
        )(x1, x1)
        # Skip connection 1.
        x2 = layers.Add()([attention_output, encoded_patches])
        # Layer normalization 2.
        x3 = layers.LayerNormalization(epsilon=1e-6)(x2)
        # MLP.
        x3 = mlp(x3, hidden_units=transformer_units, dropout_rate=0.1)
        # Skip connection 2.
        encoded_patches = layers.Add()([x3, x2])

    # Create a [batch_size, projection_dim] tensor.
    representation = layers.LayerNormalization(epsilon=1e-6)(encoded_patches)
    representation = layers.Flatten()(representation)
    representation = layers.Dropout(0.5)(representation)
    # Add MLP.
    features = mlp(representation, hidden_units=mlp_head_units, dropout_rate=0.5)
    # Classify outputs.
    logits = layers.Dense(num_classes)(features)
    # Create the Keras model.
    model = keras.Model(inputs=inputs, outputs=logits)
    return model

下圖是一個 Vision Transformer 的訓練結果，此處不加載預訓練模型，只是用于作為 Demo，