Description: Vision Transformer paper note — apply a near-vanilla Transformer encoder directly to a sequence of image patches (16×16), with a learnable [class] token and 1D position embeddings; with enough pre-training data (JFT-300M) it beats ResNet CNNs while using far less pre-training compute, showing large-scale data can substitute for the convolutional inductive bias
My Notion Note ID: K2F-B-2
Created: 2026-08-02
Updated: 2026-08-02
License: Free to share: please credit Yu Zhang and link back to yuzhang.io

1. Summary

Title: An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale Authors: A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby (Google Research, Brain Team) Paper: arXiv:2010.11929 (ICLR 2021) Github: google-research/vision_transformer

ViT (Vision Transformer) — feed a standard Transformer encoder (from Attention Is All You Need) an image, by cutting the image into fixed patches and treating each patch as a token. The design goal is deliberately minimal: change as little as possible from the NLP Transformer so its scalable architectures and implementations transfer almost out of the box.

Core question: CNNs dominate vision because convolution bakes in the right priors — locality, 2D neighborhood structure, translation equivariance. Can a Transformer, which has none of these built in, compete on images?

Key insight: yes — if you give it enough data. With mid-sized training (ImageNet alone), ViT underperforms comparable ResNets, precisely because it lacks the convolutional inductive bias and must learn spatial structure from scratch. But pre-trained on a large enough corpus (ImageNet-21k, or especially JFT-300M), ViT matches or beats the best CNNs — large-scale data substitutes for the inductive bias. This "data beats bias at scale" lesson is the paper's real contribution.

How images become tokens:

Split image into $16\times16$ patches → flatten → linear projection = patch embeddings (tokens).
Prepend a learnable [class] token (à la BERT) whose final state is the image representation.
Add learnable 1D position embeddings; feed the sequence to a plain Transformer encoder; classify from the [class] token.

Main results: pre-trained on JFT-300M, ViT-H/14 reaches 88.55% top-1 on ImageNet (90.72% ReaL, 94.55% CIFAR-100, 77.63% VTAB), beating BiT-L (big ResNet, 87.54%) and edging out Noisy Student (88.4/88.5%) — while using 2.5k vs 9.9k TPUv3-core-days of pre-training (~4× cheaper).

ViT overview: an image is split into fixed 16×16 patches, each linearly embedded; a learnable [class] token is prepended, position embeddings are added, and the sequence is fed to a standard Transformer encoder; an MLP head on the [class] token's output produces the classification.

2. Key Contributions

Vision Transformer — a nearly unmodified Transformer encoder applied directly to image patches, with a [class] token and 1D position embeddings.
The scale lesson — quantifies that ViT trails CNNs on small data but overtakes them once pre-training data is large enough (ImageNet-21k / JFT-300M), i.e. data can replace the convolutional inductive bias.
SOTA at lower cost — matches/beats top CNNs on many recognition benchmarks at a fraction of the pre-training compute, making the case for Transformers as a general vision backbone.

3. Method

3.1 Patch Embedding & Sequence Construction

Reshape image $x \in \mathbb{R}^{H\times W\times C}$ into $N$ flattened patches $x_p \in \mathbb{R}^{N\times(P^2\cdot C)}$ , where $(P,P)$ is the patch size and $N = HW/P^2$ is the number of patches = the sequence length. A trainable linear projection $E$ maps each flattened patch to the model dimension $D$ — these are the patch embeddings.

A learnable [class] token $x_\text{class}$ is prepended; its state at the encoder output ( $z_L^0$ ) is the image representation. Learnable 1D position embeddings $E_\text{pos}$ are added (2D-aware variants gave no significant gain). The full input:

z_0 = [\,x_\text{class};\ x_p^1 E;\ x_p^2 E;\ \cdots;\ x_p^N E\,] + E_\text{pos}, \qquad E \in \mathbb{R}^{(P^2\cdot C)\times D},\ E_\text{pos} \in \mathbb{R}^{(N+1)\times D}

(" $16\times16$ words": each $16\times16$ patch is a "word" token; ViT-L/16 = the Large variant with $16\times16$ patches.)

3.2 The Encoder

A standard Transformer encoder of $L$ layers, each alternating multi-head self-attention (MSA) and an MLP block, with LayerNorm before each block (pre-norm) and residual connections after:

z_\ell' = \text{MSA}(\text{LN}(z_{\ell-1})) + z_{\ell-1}, \qquad z_\ell = \text{MLP}(\text{LN}(z_\ell')) + z_\ell', \qquad \ell = 1\dots L

y = \text{LN}(z_L^0)

The MLP has two layers with a GELU nonlinearity. The classification head is an MLP with one hidden layer at pre-training, a single linear layer at fine-tuning. Model variants:

Model	Layers	Hidden $D$	MLP size	Heads	Params
ViT-Base	12	768	3072	12	86M
ViT-Large	24	1024	4096	16	307M
ViT-Huge	32	1280	5120	16	632M

3.3 Inductive Bias & Hybrid Variant

Inductive bias: ViT has far less image-specific bias than a CNN. In CNNs locality / 2D neighborhood / translation equivariance are baked into every layer. In ViT, only the MLP layers are local; self-attention is global. 2D structure enters at just two points — cutting the image into patches, and interpolating position embeddings at fine-tuning. Position embeddings start with no 2D information; all spatial relations are learned.

Hybrid: instead of raw patches, feed patches extracted from a CNN feature map (e.g. a ResNet stage) through the same projection $E$ . Special case: $1\times1$ patches = flatten the feature map's spatial dims. Hybrids help at small scale but the advantage vanishes as models/data grow.

3.4 Fine-Tuning at Higher Resolution

Pre-train large, fine-tune to downstream tasks: drop the pre-trained head, attach a zero-initialized $D\times K$ layer ( $K$ = downstream classes). Fine-tuning at higher resolution than pre-training usually helps; keeping patch size fixed makes the sequence longer, so the pre-trained position embeddings are 2D-interpolated to the new grid. This interpolation + initial patch extraction are the only manual 2D-structure injections.

4. Experiments & Results

Pre-training datasets (scale is the whole point):

ImageNet (ILSVRC-2012) — 1k classes, 1.3M images.
ImageNet-21k — 21k classes, 14M images.
JFT-300M — 18k classes, 303M images (in-house).

Transferred to ImageNet, ImageNet-ReaL, CIFAR-10/100, Oxford Pets, Oxford Flowers-102, VTAB (19 tasks). Pre-training with Adam ( $\beta_1=0.9,\beta_2=0.999$ ), batch 4096, weight decay 0.1; fine-tuning with SGD+momentum, batch 512.

vs SOTA CNNs (Table 2, top-1 %; pre-training cost in TPUv3-core-days):

	ViT-H/14 (JFT)	ViT-L/16 (JFT)	ViT-L/16 (I21k)	BiT-L (R152×4)	Noisy Student
ImageNet	88.55	87.76	85.30	87.54	88.4/88.5
ImageNet ReaL	90.72	90.54	88.62	90.54	90.55
CIFAR-100	94.55	93.90	93.25	93.51	—
VTAB (19)	77.63	76.28	72.72	76.29	—
TPUv3-core-days	2.5k	0.68k	0.23k	9.9k	12.3k

ViT-L/16 pre-trained on JFT beats BiT-L (same pre-training data) on all tasks at far less compute; ViT-H/14 pushes further on the hardest ones. Even the public ImageNet-21k pre-training is competitive.

The scale dependency (the key finding): pre-trained on ImageNet only, ViT-Large underperforms ViT-Base (large models overfit without enough data); the ordering flips as pre-training data grows — ViT needs scale to pay off. A small self-supervised experiment (masked patch prediction) gives ViT-B/16 79.9% on ImageNet, hinting at the direction later works (incl. CroCo) take.

5. Strengths / Limitations / Legacy

Strengths

Radical simplicity — reuses the NLP Transformer almost verbatim, so vision inherits its scalability and tooling.
Turns "CNNs have the right priors" into a quantitative, data-dependent statement rather than dogma.
Strong transfer at markedly lower pre-training cost; robust across many benchmarks.

Limitations

Data-hungry — without large-scale pre-training it trails CNNs; not a from-scratch-on-small-data method.
Global $O(N^2)$ attention → cost grows with resolution (more patches); high-res dense tasks are expensive.
Coarse $16\times16$ patches and 1D position embeddings discard fine spatial structure that convolutions capture cheaply.

Legacy (why it's in this reading arc)

Establishes the patch-token + [class] token + position-embedding recipe every later vision Transformer reuses.
Its global self-attention is exactly the mechanism carried into multi-view 3D: CroCo does cross-view attention between two images, and DUSt3R/VGGT extend it to global attention across many views.
The self-supervised masked-patch teaser foreshadows CroCo's cross-view completion pre-training — the bridge from ViT to the 3D reconstruction line.

References

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale. ICLR. arXiv:2010.11929, code. — source paper (Fig. 1 overview above)
Transformer: the encoder ViT applies to image patches — read first
Kolesnikov, A., et al. (2020). Big Transfer (BiT): General Visual Representation Learning. ECCV. — the large-ResNet baseline ViT is measured against
Devlin, J., et al. (2019). BERT. NAACL. — source of the prepended [class] token idea
Raghu, M., et al. (2021). Do Vision Transformers See Like Convolutional Neural Networks?. NeurIPS. — analysis of what ViT's global attention learns vs CNNs

ViT: An Image Is Worth 16×16 Words

Table of Contents