Transformer Architecture logopeterindia.net
Knowledge Hub

Transformer Architecture

How the Transformer works: self-attention, multi-head attention, positional encoding and encoder and decoder blocks, with the original paper, key variants and learning resources.

20 topics•5 categories•Updated November 18, 2026

The Original Transformer

4 entries

Start with the paper and the best visual and code walkthroughs.

Building Blocks

4 entries

Attention is wrapped in residual connections, layer normalization and positional information.

Encoder, Decoder and Beyond

4 entries

BERT uses the encoder, GPT the decoder, T5 both, and ViT applies the stack to images.

Libraries & Learning Resources

3 entries

Run, study and build Transformers.

Frequently Asked Questions

What is the Transformer architecture?

A neural network design introduced in 2017 in Attention Is All You Need that processes a whole sequence at once using attention instead of recurrence or convolution. Stacks of attention and feed-forward blocks, with residual connections and layer normalization, form the encoder and decoder.

What is self-attention?

Each token computes a weighted mix of all tokens in the sequence, with weights learned from queries and keys, so every position can use context from every other position. Multi-head attention runs several such attentions in parallel.

Why do Transformers need positional encoding?

Attention treats its input as a set with no inherent order, so position information is added. The original paper used sinusoidal encodings, and many newer models use alternatives such as rotary position embeddings.

What is the difference between encoder-only, decoder-only and encoder-decoder models?

Encoder-only models such as BERT read the whole input to build representations. Decoder-only models such as GPT-3 generate text one token at a time. Encoder-decoder models such as T5 read an input and generate an output.

How are Transformers used for images?

Vision Transformers split an image into patches, project each into a vector, add position information and a CLS token, then pass the sequence through standard Transformer layers.