How Vision Transformers (ViT) Work (2026): Patches, Attention, CLS

From Image to Tokens to a Label

11 entries

A LinkedIn post from JoinAI (26 October 2026) explains that a Vision Transformer turns an image into a sequence of tokens, where each token represents a small patch of the image, and attention lets those patches exchange information so the model can connect details across the picture. It lists five steps. First, split the image into patches; in its example a 64 × 64 image becomes sixteen 16 × 16 patches. Second, convert each patch into a vector using the same learned projection. Third, add position information and a special classification token called CLS. Fourth, pass the sequence through Transformer layers so tokens exchange information through attention. Fifth, feed the final CLS representation into a classifier to predict a label such as "cat". The post says to use ViTs for classifying images, adapting a pretrained vision model to your own image dataset, and building a visual backbone for detection or segmentation with a suitable task head. Its takeaway is that smaller patches preserve finer detail but create more tokens, increasing attention cost, so patch size and image resolution should be chosen together. This page follows those steps and lists papers, pretrained models, libraries and tutorials. Descriptions are paraphrased from each source and are not rankings; the post's link to its own course is not included.

01How do Vision Transformers work? (LinkedIn post by JoinAI)The post that explains the five ViT steps, when to use ViTs and the patch-size takeaway. 02Deep LearningDeep learning hub, on this site. 03Machine LearningMachine learning hub, on this site. 04Vision-Language ModelsModels that combine images and text, on this site. 05Multimodal Language ModelsLanguage models that accept images and other inputs, on this site. 06Multimodal Embedding ModelsEmbedding models for images and text, on this site. 07Image Data Analytics SolutionsImage analytics tools, on this site. 08OCR ModelsModels that read text from images, on this site. 09Diffusion ModelsGenerative image models, on this site. 10LLM Training and Inference OptimizationAttention cost and the KV cache, on this site. 11Looped Transformers and Shared MemoryResearch on looped Transformers, on this site.

1. Patches, Projection, Position and CLS (Steps 1 to 3)

6 entries

Split the image into patches, project each into a vector with the same learned projection, then add position information and a CLS token.

12An Image is Worth 16x16 Words (arXiv)The original Vision Transformer paper: applying a standard Transformer to sequences of image patches. 13google-research/vision_transformerGoogle Research code and models for the original ViT paper. 14Vision Transformer (Hugging Face Transformers)Documentation for ViT in the Transformers library. 15google/vit-base-patch16-224A ViT-Base model card on Hugging Face with 16 × 16 patches at 224 × 224 resolution. 16Image classification with Vision Transformer (Keras)Keras example that builds a ViT from patch extraction through classification. 17Vision Transformers Need Registers (arXiv)Paper on artifacts in ViT feature maps and extra register tokens as a fix.

2. Attention Layers and the Classifier (Steps 4 and 5)

5 entries

Transformer layers let tokens exchange information through attention, and the final CLS representation feeds a classifier.

18Attention Is All You Need (arXiv)The paper that introduced the Transformer architecture. 19Do Vision Transformers See Like Convolutional Neural Networks? (arXiv)Study comparing the internal representations of ViTs and CNNs. 20torchvision VisionTransformerPyTorch documentation for the ViT implementation in torchvision. 21Image classification task guide (Hugging Face)Hugging Face guide to fine-tuning a model for image classification. 22Deep LearningNeural network fundamentals, on this site.

3. When to Use a ViT

6 entries

Classifying images, adapting a pretrained vision model to your own dataset, or building a visual backbone for detection or segmentation with a suitable task head.

23DeiT: Training data-efficient image transformers (arXiv)Paper on training ViTs more data-efficiently, relevant when adapting to smaller datasets. 24DETR (arXiv)End-to-end object detection with Transformers, using a task head on a visual backbone. 25Object detection task guide (Hugging Face)Hugging Face guide to object detection. 26Image segmentation task guide (Hugging Face)Hugging Face guide to semantic segmentation. 27Segment Anything (arXiv)Promptable segmentation model that uses a ViT image encoder. 28segment-anything repositoryMeta's code for the Segment Anything model.

4. Patch Size, Resolution and Attention Cost

5 entries

Smaller patches preserve finer detail but create more tokens, increasing attention cost, so choose patch size and resolution together.

29Swin Transformer (arXiv)Hierarchical Vision Transformer using shifted windows to limit attention cost. 30Swin-Transformer repositoryMicrosoft's official implementation of Swin Transformer. 31Patch n' Pack: NaViT (arXiv)Vision Transformer that handles inputs of any aspect ratio and resolution. 32FlashAttention (arXiv)Fast and memory-efficient exact attention, relevant when token counts grow. 33LLM Training and Inference OptimizationAttention cost and memory bottlenecks, on this site.

5. Pretrained Models, Self-Supervision and Libraries

8 entries

Reusable backbones and libraries for ViT-based work.

34Masked Autoencoders Are Scalable Vision Learners (arXiv)Self-supervised pretraining that reconstructs masked image patches. 35DINOv2 (arXiv)Paper on learning robust visual features without supervision. 36dinov2 repositoryMeta's PyTorch code and models for DINOv2. 37CLIP (arXiv)Learning visual models from natural language supervision, using ViT image encoders. 38SigLIP (arXiv)Sigmoid loss for language-image pre-training. 39OpenCLIPOpen-source implementation of CLIP. 40timm (PyTorch Image Models)Large collection of PyTorch image models; GitHub lists v1.0.30 (22 September 2026). 41Hugging Face TransformersModel-definition framework; GitHub lists v5.22.1 (30 October 2026).

Frequently Asked Questions

What is a Vision Transformer?

A model that turns an image into a sequence of tokens, where each token represents a small patch, and uses attention so patches can exchange information, as the LinkedIn post this page follows explains.

How does a Vision Transformer work step by step?

Split the image into patches, project each patch into a vector with the same learned projection, add position information and a CLS token, pass the sequence through Transformer layers, then feed the final CLS representation into a classifier.

What is the CLS token?

A special classification token added to the patch sequence. After the Transformer layers, its final representation is fed to a classifier to predict a label such as "cat".

Why are position embeddings added?

Patches are projected into vectors without inherent order, so position information tells the model where each patch came from in the image.

What does patch size change?

Smaller patches preserve finer detail but create more tokens, which increases attention cost. The post advises choosing patch size and image resolution together.

When should I use a ViT?

For classifying images, adapting a pretrained vision model to your own dataset, or building a visual backbone for detection or segmentation with a suitable task head.

How is a ViT different from a CNN?

A ViT uses attention over patch tokens rather than convolutions. A study linked on this page compares what the two learn internally. Which is better depends on data, compute and task.

How current is this page?

Links and release facts were checked on 1 November 2026. LinkedIn may ask visitors to sign in. Descriptions follow each source's own pages and are not rankings or guarantees.