Based on An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
1Built-in assumptions
Convolutional networks long dominated image recognition. They build in two useful assumptions: nearby pixels are related, and a pattern means the same thing wherever it appears in the image. Such built-in assumptions are called inductive biases. Transformers, which had succeeded in language, have far fewer of them: their attention mechanism can relate any part of the input to any other from the start. The question was whether a standard Transformer, with as few changes as possible, could classify images.
2Patches as words
The Vision Transformer splits an image into fixed-size square patches, for example 16 by 16 pixels. A 224-by-224 image gives a 14-by-14 grid, or 196 patches. Each patch is flattened into a list of its pixel values, 16 × 16 × 3 = 768 numbers for a color image, and a learned linear projection turns it into a vector: one token, like a word in a sentence. Learned position embeddings are added so the model knows where each patch came from. A special classification token is added to the sequence, and its final vector is used to predict the image’s class.
3Learning what convolutions assume
Because attention can connect any two patches from the first layer, the model must learn from data which relationships matter, including the importance of nearby patches that a convolutional network gets for free. The paper found that some attention heads look across most of the image even in the lowest layers, while others stay local. The learned position embeddings came to reflect the image’s two-dimensional layout: patches in the same row or column ended up with similar embeddings.
4Data makes the difference
Trained only on ImageNet, about 1.3 million labeled images, the Vision Transformer was a few percentage points less accurate than convolutional ResNets of similar size. Pretrained on much larger datasets, 14 million or 300 million images, and then fine-tuned on the smaller target dataset, an approach called transfer learning that reuses the general visual features learned first, it matched or exceeded the best convolutional networks on several benchmarks while needing substantially less computation to pretrain. The paper’s summary is that large-scale training can outweigh built-in assumptions.
5Trade-offs
Datasets of hundreds of millions of labeled images are not available to most teams; the largest one used in the paper is private. With limited data, a convolutional network’s built-in assumptions remain an advantage. The number of patches also grows with image resolution, and attention’s cost grows with the square of the number of patches. The paper focused on classification and left tasks such as detecting and outlining objects to future work.
Alexey Dosovitskiy et al. · 2020. Adapted by DailyPaper; illustrative examples are labeled in the reading.