PART 1 • FOUNDATIONS • LEVEL 03
Linear Algebra for Machine Learning
See data as geometry. Build intuition for vectors, matrices, projections and decompositions—the language behind regression, embeddings, neural networks, PCA and modern AI.
vAv
TRANSFORMATIONA = [ 2 1 ]
[ 1 2 ]v → Av
[ 1 2 ]v → Av
LEARNING OUTCOMES
By the End of This Level, You Can
QUICK REVISION
Five Ideas to Recall Before Linear Algebra
These familiar ideas become the foundation of vector and matrix reasoning.
A scalar has size and may indicate direction through its sign.
A point such as (3, 2) describes location along named axes.
Equations express constraints between quantities.
A transformation maps one vector space into another.
Shape determines whether operations are compatible.
CONCEPT 01
Scalar, Vector, Matrix and Tensor
The number of axes changes, but every structure stores numerical information that an AI system can transform.
7.5One value: a learning rate, bias or loss.
[2, 4, 6]One sample, feature set or embedding.
[[1, 2], [3, 4]]A dataset, weight table or image channel.
(batch, height, width, channel)Batches, images, sequences and model activations.
Shape-first habit: Before any operation, write the shape of every input and predict the shape of the output.
A scalar is one value, a vector is an ordered list, a matrix is a two-dimensional grid, and a tensor generalizes the idea to more axes. Their meaning depends on context: a vector may represent one sample, parameters or an embedding; a matrix may represent a dataset or transformation.
A batch of 32 RGB images of size 224 by 224 is commonly represented by a rank-four tensor with shape (32, 3, 224, 224).
Shape annotations make model equations and neural-network code much easier to verify.
Tensor rank in linear algebra and rank as the number of independent matrix directions are different concepts.
CONCEPT 02
Vectors Carry Magnitude and Direction
A vector can represent geometry or an ordered collection of features.
[2, 1] + [1, 3] = [3, 4]
Combine matching components. Shapes must agree.
3[2, 1] = [6, 3]
Stretch magnitude; a negative scalar also reverses direction.
au + bv
Build a new vector from weighted directions.
A vector can be understood as coordinates, an arrow or a list of features. Addition combines corresponding components, and scalar multiplication changes magnitude and possibly direction. Linear combinations build new vectors from weighted existing directions.
Combining topic embeddings with weights creates a new vector representing a document’s mixture of topics.
Model parameters and feature vectors interact through linear combinations throughout ML.
Vector components only become meaningful after the coordinate system, feature order and units are defined.
CONCEPT 03
Norms and Distance
A norm measures vector size. A distance measures how far two vectors are apart.
For [3, −4], L1 = 7. Robust geometry with diamond-shaped boundaries.
For [3, −4], L2 = 5. The usual straight-line length.
Used by k-NN, clustering and similarity-based systems.
Scaling matters: A large-range feature can dominate Euclidean distance. Standardize meaningful numeric features before distance-based learning.
A norm measures vector size. The L1 norm sums absolute values, the L2 norm gives Euclidean length, and the max norm uses the largest absolute component. Distance applies a norm to the difference between two vectors, so feature scale directly affects which points appear close.
Without scaling, a salary feature measured in thousands may dominate an age feature when computing nearest neighbours.
Norms appear in distance-based models, regularization, optimization and error measurement.
Do not choose Euclidean distance automatically for high-dimensional or mixed-type data.
CONCEPT 04
Dot Product: Alignment as a Number
Multiply matching components and add them. The result is a scalar.
[2, 3]·[4, 1]
(2 × 4) + (3 × 1) = 11x · y = ‖x‖ ‖y‖ cos θ
The dot product multiplies corresponding components and sums them. Geometrically it equals the product of both lengths and the cosine of their angle, so it measures directional alignment while also depending on magnitude.
A linear model score is the dot product between a feature vector and learned weights, followed by a bias.
Attention scores, linear layers and similarity calculations rely on dot products.
A large dot product may result from large norms rather than genuinely similar direction.
CONCEPT 05
Cosine Similarity Ignores Overall Scale
Normalize the dot product by both vector lengths to compare direction.
cosine(x, y)=
x · y‖x‖₂ ‖y‖₂
Compare query and document embeddings.
Match user and item profiles by direction.
Group directionally similar high-dimensional vectors.
Cosine similarity divides the dot product by both L2 norms, leaving a value from -1 to 1 that compares direction. It is especially useful when magnitude reflects document length or activity level rather than semantic meaning.
Two documents with proportional word-frequency vectors receive cosine similarity 1 even if one is much longer.
Embedding search often ranks items by cosine similarity after normalization.
Cosine similarity is undefined for a zero vector; handle empty or all-zero representations explicitly.
CONCEPT 06
Matrix Operations and Shape Rules
Some operations are elementwise; multiplication composes relationships.
(m, n) ± (m, n) → (m, n)Combine corresponding elements.
A * BMultiply cells at identical positions.
Aᵀ: (m, n) → (n, m)Rows become columns.
(m, n) @ (n, p) → (m, p)Inner dimensions match and disappear.
Matrix addition and elementwise multiplication combine matching cells, whereas matrix multiplication composes relationships between spaces. Transpose swaps axes. Shape rules are the contract: addition requires compatible dimensions, while matrix multiplication requires matching inner dimensions.
A data matrix (samples, features) multiplied by weights (features, outputs) produces predictions (samples, outputs).
Reading dimensions before values prevents many model implementation errors.
In NumPy, * is elementwise multiplication and @ is matrix multiplication.
CONCEPT 07
Matrix Multiplication, Cell by Cell
Each output cell is the dot product of one row from A and one column from B.
A • 2 × 3
×123456
B • 3 × 2
→789101112
C • 2 × 2
5864139154
C[0, 0] = 1×7 + 2×9 + 3×11 = 58Work: O(mnp) for the classical triple-loop algorithm • Result space: O(mp)Each cell of a matrix product is one row–column dot product. The inner dimension indexes the features being combined; the outer dimensions determine the result shape. Matrix multiplication is associative but generally not commutative.
For A with shape (2, 3) and B with shape (3, 4), the result has shape (2, 4) and each cell sums three products.
Batched predictions and neural-network layers are repeated matrix products.
A @ B and B @ A may have different shapes, values, or one direction may be invalid.
CONCEPT 08
Matrices as Linear Transformations
A matrix can rotate, scale, reflect or shear every vector while preserving linear combinations.
[[sₓ, 0], [0, sᵧ]][[cosθ, −sinθ], [sinθ, cosθ]][[1, k], [0, 1]][[-1, 0], [0, 1]]A matrix transformation maps every input vector into an output vector. Scaling, rotation, reflection and shear are linear because they preserve addition and scalar multiplication. The columns of the matrix show where the standard basis vectors move.
A 2×2 matrix transforms a square grid into a parallelogram, revealing direction and area changes.
Learned weight matrices transform representations between neural-network layers.
Translation is not a linear transformation in ordinary coordinates because it moves the zero vector.
CONCEPT 09
Systems, Determinant and Inverse
The system Ax = b asks which input vector becomes b after transformation A.
Signed area or volume scaling. If det(A) = 0, dimensions collapse and A is singular.
Reverses an invertible transformation. It exists only for square full-rank matrices.
Numerically prefer a solver over explicitly computing A⁻¹b.
Solving Ax=b asks which input maps to b. The determinant measures signed area or volume scaling for a square matrix. A zero determinant means at least one direction collapsed, so a unique inverse does not exist. Numerical solvers use stable factorizations instead of constructing the inverse directly.
Two proportional equations describe the same line, making the coefficient matrix singular and the solution non-unique or inconsistent.
Least-squares solvers handle systems that are overdetermined or do not have an exact solution.
A nonzero but tiny determinant does not guarantee numerical stability; inspect conditioning.
CONCEPT 10
Span, Independence, Basis and Rank
These ideas reveal how much distinct information a collection of vectors contains.
The set formed by every linear combination of given vectors.
Only the all-zero coefficients produce the zero vector.
An independent set that spans the complete space.
Dimension of the column space; low rank means redundancy.
Span is the set of vectors reachable through linear combinations. Independence means no vector can be constructed from the others. A basis is an independent spanning set, and rank counts independent directions. Rank therefore measures the effective information carried by a matrix.
If two dataset columns are exact multiples, they add no new direction and reduce the independent feature rank.
Low-rank approximations compress data and multicollinearity destabilizes coefficients.
A large number of columns does not imply an equally large amount of independent information.
CONCEPT 11
Orthogonality and Projection
Orthogonal vectors have zero dot product. Projection extracts the component of one vector along another.
uvprojᵤ(v)
projᵤ(v) = (v·u / u·u)u
- Least squares projects targets onto the column space.
- Orthogonal directions reduce interference and simplify calculations.
- Residuals are perpendicular to fitted directions at the optimum.
Orthogonal vectors have zero dot product and do not share a component. Projection finds the closest point along a direction or subspace. In least squares, predictions are the projection of the target vector onto the feature column space, leaving an orthogonal residual.
Projecting [3,4] onto [1,0] keeps the horizontal component [3,0].
Projection explains least squares, PCA and decomposition methods geometrically.
The projection formula divides by u·u, so the direction vector must not be zero.
CONCEPT 12
Eigenvectors and Eigenvalues
An eigenvector keeps its direction under a transformation; the eigenvalue tells how it is scaled.
Av = λvA
transformation
vspecial direction
λscale factor
An eigenvector keeps its direction after transformation, while its eigenvalue gives the scaling and possible reversal. Eigenvectors reveal natural axes of a transformation. For a covariance matrix, the largest eigenvalues identify directions with the greatest variance.
PCA rotates data toward covariance eigenvectors and keeps the directions carrying the most variation.
Spectral clustering, stability analysis and PCA all use eigenstructure.
Not every matrix has a full set of real eigenvectors; interpretation depends on the matrix type.
CONCEPT 13
Singular Value Decomposition
SVD decomposes any matrix into orthogonal directions and non-negative strengths.
A=UΣVᵀ
Output-space directions: left singular vectors.
Singular values ordered by captured strength.
Input-space directions: transposed right singular vectors.
Keep top components for compression, denoising and latent semantics.
Singular value decomposition factors any matrix into input directions, non-negative strengths and output directions: A=UΣVᵀ. Keeping only the largest singular values produces the best low-rank approximation under common norms and separates dominant structure from weaker detail.
A document-term matrix can be approximated with fewer latent dimensions for search and compression.
SVD supports PCA, pseudoinverses, denoising and recommendation systems.
Discarded components may contain rare but important signals, so rank selection needs validation.
WHY ML NEEDS THIS
Linear Algebra Inside Real Models
Rows are samples; columns are features.
Weighted feature combinations produce outputs.
Weights transform activations between spaces.
Dot products measure query–key alignment.
Low-rank structure reduces storage and noise.
Distance quantifies prediction error.
INTERACTIVE LEARNING • CODEBHAVYA PREMIUM VISUALIZER
LIVE🎬 Matrix Transformation — Visual Flow
Follow v = [2, 1] through A = [[2, 1], [1, 2]] until it becomes Av = [5, 4].
STEP 1 OF 7
Read the input vector
Begin with v = [2, 1] in the input space.
Step 1 of 7
PROGRAM TRACING • TRUE LOOP FLOW
Trace Matrix Multiplication
Watch the cursor revisit all three loop lines while every row–column product updates one result cell.
Press Next to execute the first statement.
EXPRESSION
—RESULT MATRIX / OUTPUT
[[0, 0], [0, 0]]
Step 0 of 32
CORE ALGORITHM PATTERN
Matrix Multiplication from Scratch
The loops select a row, select a column and accumulate their dot product.
A = [[1, 2], [3, 4]]
B = [[5, 6], [7, 8]]
result = [[0, 0], [0, 0]]
for i in range(2):
for j in range(2):
for k in range(2):
result[i][j] += A[i][k] * B[k][j]
print(result)CELL RULE
Cᵢⱼ = Σₖ AᵢₖBₖⱼ- i: chooses a row from A.
- j: chooses a column from B.
- k: walks through their matching components.
- Compatibility: A columns = B rows.
- Complexity: O(mnp) for m×n multiplied by n×p.
PROGRAMMING PROBLEMS • CODEBHAVYA PATTERN
Practise Linear Algebra with Python
Attempt each problem independently. The checker rewards correct reasoning, required operations and exact output.
0 / 5Solved independently0 / 500Best score
QUICK KNOWLEDGE CHECK
Test Your Geometric and Matrix Reasoning
Select one answer for every question. Results show your answer, the correct answer and a clear explanation.
Not checked yet
PLACEMENT CHECKPOINT
How Linear Algebra Appears in Hiring Rounds
Strong candidates connect calculation, geometry, dimensions, numerical stability and model behaviour.
Shape Reasoning
Predict valid products and output dimensions without running code.
Manual Calculation
Compute dot products, norms and a small matrix product accurately.
Model Connection
Explain how a linear layer, PCA or cosine search uses these operations.
Edge Cases
Discuss singular matrices, zero vectors, low rank and unstable inversion.
CodeBhavya interview pattern:Define it → Give the equation → Explain the geometry → Connect it to ML → State one edge case.
INTERVIEW PREPARATION
🎤 Linear Algebra for ML — Interview Questions
Answer aloud before selecting Show Answer for each explanation.
KEY TAKEAWAY
Linear Algebra in One View
Vectors and matrices organize features, weights and activations.
Norms and dot products measure size, distance and alignment.
Matrix products map data into useful spaces.
Eigenvectors and SVD expose dominant structure.
Machine learning becomes clearer when every formula is read twice: once as arithmetic and once as geometry.
EXTRA TIPS
Habits for Reliable Matrix Reasoning
Annotate every matrix with its shape before multiplying.
Read A @ B as rows of A meeting columns of B.
Use np.linalg.solve instead of explicitly computing an inverse.
Check for a zero vector before cosine normalization.
Use matrix_rank and condition numbers when systems behave unstably.
Verify hand calculations with np.allclose, not exact float equality.
EXTRA PRACTICE QUESTIONS
Strengthen Linear Algebra Thinking
Predict the result and geometry first, then verify with NumPy.
- 01
Add [2, −1, 4] and [3, 5, −2].
- 02
Compute the L1 and L2 norms of [6, 8].
- 03
Find the Euclidean distance between [1, 2] and [4, 6].
- 04
Calculate the dot product of [1, 3, 2] and [4, −1, 5].
- 05
Determine whether [2, 1] and [−1, 2] are orthogonal.
- 06
Predict the output shape of (8, 5) @ (5, 3).
- 07
Explain why (4, 2) @ (3, 4) is invalid.
- 08
Multiply [[1, 0], [2, 1]] by [3, 4].
- 09
Find the determinant of [[2, 1], [4, 2]] and interpret it.
- 10
Project [3, 4] onto the x-axis vector [1, 0].
- 11
Explain what a rank-one dataset matrix says about its features.
- 12
Describe how truncated SVD can compress an image or document matrix.