Linear Algebra for Artificial Intelligence logopeterindia.net
Knowledge Hub

Linear Algebra for Artificial Intelligence

How linear algebra connects to AI — the arithmetic of vectors, matrices and tensors behind data, neural networks, embeddings, attention and efficient fine-tuning.

66 topics•8 categories•Updated October 2026

Why Linear Algebra Matters for AI

3 entries

Almost everything a modern AI system computes is linear algebra. Data are stored as vectors, matrices and tensors; a neural-network layer is a matrix multiplication followed by a nonlinearity; word and image embeddings are vectors compared by dot products or cosine similarity; and transformer attention is a sequence of matrix products. Decompositions such as SVD and eigendecomposition power PCA, recommender systems and model compression, while low-rank updates make fine-tuning large models affordable. GPUs and libraries such as BLAS exist largely to make matrix multiplication fast. This page collects free books, courses, papers and tools for each area.

01Linear algebraOverview of vector spaces, linear maps and matrices. 02Matrix multiplicationThe core operation behind neural-network layers and attention. 03TensorMulti-dimensional arrays and their algebra, the data structure of deep learning.

Foundations & Learning Resources

11 entries

Core ideas and where to learn them.

04Linear Algebra (MIT OCW 18.06, Gilbert Strang)The widely used free university course. 05Linear Algebra Done Right (Axler)Proof-based textbook, free to download. 06Essence of Linear Algebra (3Blue1Brown)Visual video series building geometric intuition. 07Mathematics for Machine LearningFree book covering linear algebra, analytic geometry and decompositions. 08Linear Algebra and Learning from Data (Strang)Book linking linear algebra to deep learning. 09Introduction to Linear Algebra for Applied Machine Learning with PythonFree online notes with Python examples. 10Linear Algebra and Its Applications (Khan Academy)Gentle introduction with exercises. 11Linear Algebra (Deep Learning book, ch. 2)Concise chapter written for deep learning readers. 12The Matrix CookbookReference of matrix identities and derivatives. 13Matrix Calculus for Deep LearningGradients of vector and matrix expressions. 14Immersive Linear AlgebraInteractive online textbook.

Matrix Decompositions & Dimensionality Reduction

9 entries

Factorizations that reveal structure in data.

15Singular value decompositionFactors any matrix into rotations and scalings; basis of low-rank approximation. 16Eigenvalues and eigenvectorsDirections a linear map only scales. 17Principal component analysisDimensionality reduction by projecting onto top singular directions. 18Matrix factorization (recommender systems)Latent-factor models for recommendations. 19QR decompositionOrthogonal factorization used for least squares and eigenvalue algorithms. 20Cholesky decompositionFactorization of positive-definite matrices, used in Gaussian models. 21Low-rank approximationApproximating a matrix by one of smaller rank. 22Finding Structure with Randomness (randomized SVD)Fast probabilistic algorithms for matrix decompositions. 23scikit-learn: decomposing signals into componentsPCA, NMF and related methods in Python.

Linear Algebra Inside Neural Networks

9 entries

Where matrices and tensors appear in models.

24Attention Is All You NeedTransformer paper; attention is built from matrix products and softmax. 25The Illustrated TransformerVisual walkthrough of the matrix operations in a transformer. 26Efficient Estimation of Word Representations in Vector SpaceWord2vec embeddings and vector arithmetic. 27GloVe: Global Vectors for Word RepresentationEmbeddings learned from a word co-occurrence matrix. 28Deep Residual Learning for Image RecognitionResNet; layers as linear maps with skip connections. 29Batch NormalizationNormalization of layer inputs. 30Backpropagation (matrix form), Neural Networks and Deep Learning ch. 2Matrix-based derivation of backpropagation. 31Dive into Deep Learning: linear algebraExecutable notes on tensors and linear algebra. 32Cosine similarityMeasure used to compare embeddings.

Low-Rank Methods & Efficient AI

7 entries

Using structure in matrices to cut cost.

33LoRA: Low-Rank Adaptation of Large Language ModelsFine-tunes with low-rank updates to weight matrices. 34QLoRAFine-tuning with a quantized base model and low-rank adapters. 35Hugging Face PEFTLibrary for LoRA and other parameter-efficient methods. 36GaLore: Memory-Efficient LLM Training by Gradient Low-Rank ProjectionLow-rank projection of gradients during training. 37FlashAttentionIO-aware exact attention that reduces memory traffic. 38LinformerSelf-attention with linear complexity using low-rank projections. 39Sparse matrixStorage and algorithms for matrices that are mostly zeros.

Numerical Linear Algebra & Hardware

8 entries

Computing with matrices accurately and fast.

40Numerical Linear Algebra (Trefethen & Bau)Standard text on the algorithms and their stability. 41Matrix Computations (Golub & Van Loan)Reference on matrix algorithms. 42Fast.ai Computational Linear AlgebraCourse focused on practical algorithms. 43BLAS (Basic Linear Algebra Subprograms)Standard interface for vector and matrix kernels. 44LAPACKLibrary for decompositions and linear systems. 45NVIDIA cuBLASGPU-accelerated BLAS. 46NVIDIA CUTLASSCUDA templates for matrix multiplication and tensor-core kernels. 47Tensor Cores (NVIDIA)Hardware units for mixed-precision matrix math.

Libraries & Tools

9 entries

Software for linear algebra in Python, Julia and more.

48NumPy linear algebraMatrix and decomposition routines. 49SciPy linalgExtended linear algebra and sparse routines. 50PyTorchTensor library with linear algebra operations and autograd. 51JAXComposable NumPy-style array programming on accelerators. 52einopsReadable tensor reshaping and reductions. 53SymPy matricesSymbolic linear algebra in Python. 54Julia LinearAlgebraStandard library for linear algebra in Julia. 55EigenC++ template library for linear algebra. 56SageMathOpen-source system with exact and numeric linear algebra.

Related Pages

10 entries

Neighbouring pages on this site.

57Linear AlgebraLinear algebra web resources. 58MathematicsMathematics resources. 59Probability Theory for AIProbability for AI. 60Number Theory for AINumber theory for AI. 61Machine LearningMachine learning hub. 62Deep LearningDeep learning hub. 63Generative AIGenerative AI hub. 64Python Libraries for Data ScientistsPython data science libraries. 65AlgorithmsAlgorithms hub. 66Artificial IntelligenceAI hub.

Frequently Asked Questions

Why is linear algebra important for AI?

Data, model parameters and activations are stored as vectors, matrices and tensors, and training and inference are dominated by matrix multiplication. Understanding linear algebra explains embeddings, attention, dimensionality reduction and why GPUs speed up deep learning.

Which linear algebra topics should I learn first for machine learning?

Vectors and dot products, matrices and matrix multiplication, linear systems, eigenvalues and eigenvectors, and the singular value decomposition. Then add norms, orthogonality, projections and matrix calculus for understanding gradients.

How is linear algebra used in transformers and LLMs?

Token embeddings are vectors; attention projects them with learned weight matrices into queries, keys and values, scores them with matrix products and a softmax, and mixes the values. Feed-forward layers are further matrix multiplications.

What is LoRA and how does it relate to linear algebra?

LoRA fine-tunes a large model by freezing the original weights and learning a low-rank update, the product of two small matrices. It relies on the idea that useful weight changes can be well approximated by a matrix of low rank.

What is the difference between SVD and PCA?

PCA finds the directions of greatest variance in centred data and can be computed from the SVD of the data matrix. SVD is the more general factorization and also applies to matrices that are not centred or are not data tables.

How current is this page?

Links were checked in October 2026. This is an introductory map, not a textbook: it points to free books, courses, papers and libraries. Sites that block automated checks are kept and may need a manual click.