PART 3 • UNSUPERVISED & APPLIED ML • LEVEL 14
Dimensionality Reduction
Preserve the strongest signal while using fewer coordinates. Build geometric intuition for PCA and SVD, understand supervised and nonlinear projections, and learn when compression improves—or damages—an ML system.
COMPRESSION GOALretain signal • remove redundancy
Z = XWₖLEARNING OUTCOMES
By the End of This Level, You Can
QUICK REVISION
Six Ideas Behind Useful Compression
Dimensionality reduction changes representation while attempting to retain information needed for a task.
Each feature adds an axis to the representation space.
Correlated features may describe nearly the same direction.
Transform observations onto selected basis directions.
PCA preserves directions with the greatest sample variation.
Low effective rank permits accurate compression with fewer factors.
Compression is useful only when retained information supports the goal.
CONCEPT 01
High Dimensions Create Statistical and Computational Pressure
More features can add signal, but irrelevant, sparse or redundant coordinates make learning harder.
In sparse high-dimensional spaces, pairwise distances can become less informative.
The volume of the feature space grows rapidly while available observations remain limited.
Additional parameters and split candidates increase variance when data is insufficient.
Training, storage, similarity search and inference all process more values.
The curse of dimensionality is not simply “many columns are bad.” It describes how geometric intuition and data coverage change as dimensions grow. A fixed number of observations occupies a progressively smaller fraction of the possible space, neighbourhoods become sparse and flexible models receive more opportunities to follow random variation. Reduction can improve speed, visualization and generalization, but removing a weak-looking feature can also destroy rare or nonlinear signal. The correct question is whether a lower-dimensional representation preserves information required by the downstream task.
A 10-bin grid needs 100 cells in 2-D but one million cells in 6-D.
Connect dimensionality to sample size, distance quality, variance and computation.
Dimensionality reduction does not automatically improve every model.
CONCEPT 02
PCA Rotates Data Toward Maximum-Variance Directions
Principal Component Analysis creates orthogonal linear combinations ordered by captured variance.
Centre every feature; standardize when scale should not define importance.
Build the covariance matrix or apply SVD directly to centred data.
Eigenvectors define principal axes; eigenvalues quantify their variance.
Keep a justified number of leading directions.
Compute lower-dimensional scores Z = XWₖ.
PCA first moves the coordinate origin to the feature means. It then searches for a unit direction whose projected values have maximum variance. The second component maximizes remaining variance while being perpendicular to the first, and later components follow the same rule. Each component is a weighted combination of original features. The signs of an eigenvector may flip without changing the component, so interpretation should focus on relative loadings and direction rather than expecting one fixed sign.
A long diagonal point cloud can be summarized by position along its diagonal and a small perpendicular residual.
State that PCA is unsupervised, linear and variance-preserving—not label-aware.
The first principal component is not necessarily the most predictive direction.
CONCEPT 03
Covariance, Eigenvectors and Eigenvalues Give PCA Its Geometry
The covariance matrix records how centred features vary together.
Xc = X − μSubtract training means so covariance describes variation around the origin.
C = XcᵀXc / (n−1)A symmetric matrix whose diagonal stores feature variances.
Cv = λvDirection v retains its orientation under C and is scaled by λ.
Z = XcVₖScores describe each observation in the selected principal basis.
Positive covariance means two centred features tend to move together; negative covariance means they move oppositely. Because the covariance matrix is symmetric, its eigenvectors can be chosen orthonormal. Sorting eigenvalues from largest to smallest sorts components by explained variance. Standardization changes the question from “which raw units vary most?” to “which standardized patterns vary most?” Fit the means and scales on training data only, then reuse them for validation, test and production observations.
Duplicated features create a strong shared direction and almost no variance in their difference direction.
Be ready to calculate a 2×2 covariance matrix and interpret its off-diagonal sign.
Centred PCA and standardized PCA can produce different components.
CONCEPT 04
Explained Variance and Reconstruction Control the Trade-off
Keeping fewer components saves resources but discards part of the original representation.
λⱼ / ΣλShare retained by one componentCumulative ratios summarize total variance preserved by the first K components.
X̂ = ZVₖᵀ + μReturn to feature spaceThe difference X − X̂ measures information excluded by the selected subspace.
CV score vs KMeasure the real objectiveSelect K inside a pipeline using validation performance, latency and stability.
A 95% cumulative variance rule is a heuristic, not a universal target. Variance can belong to nuisance factors, and low-variance directions can carry class separation. Reconstruction error answers how faithfully PCA can rebuild the input, while downstream cross-validation answers whether the compressed representation supports prediction. Plot both when possible. For deployment, also consider model size, latency, numerical stability and how easily component loadings can be explained.
Two correlated measurements may need one component for accurate reconstruction, while two independent signals need both.
Choose K using cumulative variance plus task validation, never percentage alone.
Retaining 95% variance does not mean retaining 95% predictive accuracy.
PREMIUM COMPUTATIONAL VISUALIZER
CodeBhavya • Calculate, Project, Rebuild📐 PCA Geometry & Reconstruction Laboratory
Run the actual PCA pipeline. Change geometry, preprocessing and component count; inspect covariance, eigenvectors, projections, explained variance and reconstruction error.
CONCEPT 05
SVD Factorizes a Matrix into Directions and Strengths
Singular Value Decomposition works directly with rectangular matrices and underlies many PCA implementations.
X = UΣVᵀOrthonormal patterns across observations or rows.
Non-negative strengths ordered from largest to smallest.
Orthonormal feature or column directions.
SVD exists for any real m×n matrix. Keeping the first K singular triplets gives Xₖ = UₖΣₖVₖᵀ, the best rank-K approximation under Frobenius or spectral norm. For centred data, PCA directions are the columns of V and covariance eigenvalues equal squared singular values divided by n−1. Practical implementations often prefer SVD because it avoids explicitly forming the covariance matrix, which can magnify numerical error.
A repeated image pattern can be represented by a few row and column factors instead of every pixel independently.
Explain the shapes: U is m×r, Σ is r×r and Vᵀ is r×n in compact SVD.
SVD and eigendecomposition are related but not identical operations.
CONCEPT 06
Low-Rank Approximation Compresses Structured Matrices
Rank selection keeps dominant factors while treating weaker factors as detail or noise.
m × nEvery matrix value is stored.
k(m+n+1)Store K left vectors, values and right vectors.
Σⱼ₍₍>k₎₎ σⱼ²Squared singular values quantify lost Frobenius energy.
Low-rank approximation is effective when a matrix contains repeated or correlated structure. Images, document-term matrices and user-item interactions often have a smaller latent structure than their raw size suggests. Increasing K improves reconstruction monotonically but reduces compression. A visually clean reconstruction can still erase rare details, and a recommender’s reconstruction error may not match ranking quality, so evaluate against the actual task.
A smooth gradient has low effective rank; independent random pixels require many singular factors.
Know the Eckart–Young result: truncated SVD is the optimal rank-K approximation for common matrix norms.
Higher compression does not always remove only noise.
LOW-RANK COMPUTATION ENGINE
CodeBhavya • See What Rank Preserves🧩 SVD Matrix Compression Laboratory
Factorize a real matrix, retain selected singular components and compare original, reconstructed and absolute-error heatmaps.
CONCEPT 07
LDA Uses Labels to Preserve Class Separation
Linear Discriminant Analysis projects data by maximizing between-class separation relative to within-class spread.
Uses feature variance and ignores the target labels.
maximize projected varianceUses class labels to favour directions separating class means.
maximize between / within scatterWith C classes, discriminant information lives in no more than C−1 directions.
k ≤ min(d, C−1)PCA may preserve a high-variance direction that contains little class information. LDA instead constructs within-class and between-class scatter matrices, then finds directions with strong class-mean separation and low within-class dispersion. LDA must be fitted only inside training folds because it uses labels. Classical LDA also relies on distributional and covariance assumptions, and singular within-class scatter requires regularization or dimensionality control.
Two long horizontal classes separated vertically may have PC1 horizontally but their best discriminant direction vertically.
The most important PCA–LDA difference is unsupervised variance versus supervised separation.
LDA dimensionality reduction is different from Latent Dirichlet Allocation.
CONCEPT 08
t-SNE and UMAP Are Powerful Visualization Tools—Not Proof of Clusters
Nonlinear embeddings emphasize neighbourhood structure and are usually used for exploration in two or three dimensions.
Matches local similarity distributions; perplexity influences the effective neighbourhood scale.
Builds a fuzzy local-connectivity graph and optimizes a low-dimensional layout.
Gap size, island area and visual density are not automatically meaningful.
Check seeds, settings, labels only for interpretation, and original-space evidence.
Both methods create nonlinear layouts where nearby points often remain neighbours, but the axes have no simple original-feature meaning. t-SNE can form visually separated islands even for continuous data; UMAP may preserve more broad structure in some datasets but still distorts distances. Fit-transform behaviour for new data differs by implementation. Never train a business clustering rule solely from an attractive 2-D plot. Use embeddings to generate hypotheses, inspect neighbourhoods and communicate patterns that survive checks in the original representation.
Rotating or stretching a nonlinear embedding may not change its neighbourhood meaning, while island spacing can change across runs.
Mention stochasticity, hyperparameter sensitivity and the danger of interpreting global geometry.
Visible islands are not automatic evidence of natural classes.
CONCEPT 09
Reduction Must Live Inside a Leakage-Safe Pipeline
Every learned transformation belongs to the model-selection process.
Computing scaling or PCA on the full dataset allows validation observations to influence means, variances and principal directions. This is unsupervised leakage: no labels are used, yet information from the held-out distribution has entered training. Place preprocessing, reduction and estimator in one pipeline and tune component count inside cross-validation. Store the entire fitted pipeline for inference. Monitor original features, transformed component scores and downstream performance because a stable predictor can still receive a drifting representation.
A test-set outlier can rotate a globally fitted principal component even before the model sees its target.
Interview-ready order: split → fit transforms on train → transform validation/test → evaluate.
Unsupervised preprocessing can still leak validation information.
INTERACTIVE LEARNING
🎬 Dimensionality Reduction — Visual Flow
Move from a high-dimensional objective to a validated compact representation.
Visualization, denoising, speed or prediction?
Split first; fit transformations on training data.
Linear variance, supervised separation or neighbourhood view.
Compare variance, error, stability and validation score.
Inspect loadings, lost signal and drift.
PROGRAM TRACING • TRUE NESTED-LOOP EXECUTION
Trace PCA from First Principles
Follow feature means, centering, covariance accumulation, power iteration and projection. The cursor returns through every loop exactly as Python executes.
Press Next to evaluate the first statement.
EXPRESSION
—OUTPUT
Waiting for print(...)
Step 0
CORE ALGORITHMS
Reduction Logic Before Libraries
Use these compact procedure maps for revision, coding and interviews.
- Split data and fit centering or scaling on training rows.
- Compute the centred covariance matrix.
- Find covariance eigenvalues and eigenvectors.
- Sort eigenpairs by descending eigenvalue.
- Project with the first K eigenvectors.
- Centre the training matrix Xc.
- Compute Xc = UΣVᵀ.
- Use the first K columns of V as components.
- Calculate scores XcVₖ or UₖΣₖ.
- Reuse the fitted mean and components for new rows.
- Factorize the sparse or dense matrix.
- Order singular values from largest to smallest.
- Retain K singular triplets.
- Reconstruct or use factor scores as features.
- Validate K against the downstream objective.
- Use labelled training rows only.
- Compute class means and the global mean.
- Build within-class and between-class scatter.
- Solve the generalized eigenproblem.
- Keep at most C−1 discriminant directions.
PROGRAMMING PROBLEMS • CODEBHAVYA PATTERN
💻 Dimensionality Reduction Challenges
Attempt each program independently. Workspaces, hints and model programs remain collapsed initially.
0 / 5Solved independently0 / 500Best score
QUICK KNOWLEDGE CHECK
Test Your Reduction Reasoning
Select one answer per question. Results show your choice, the correct answer and a clear explanation.
Not checked yet
PLACEMENT CHECKPOINT
Choose Reduction Like an ML Engineer
Start from the task and validation plan, then justify the representation.
Use scaled or centred PCA and validate component count inside the pipeline.
Consider TruncatedSVD so centering does not destroy sparsity.
Use LDA inside training folds when class-separation assumptions are suitable.
Use t-SNE or UMAP cautiously; compare seeds and original-space neighbours.
Use truncated SVD and evaluate reconstructed detail plus task performance.
Inspect PCA loadings, sign ambiguity, stability and reconstruction examples.
CodeBhavya interview pattern:State purpose → Identify redundancy → Split data → Fit preprocessing → Choose method → Select K → Validate lost signal → Package transformation → Monitor drift.
INTERVIEW PREPARATION
🎤 Dimensionality Reduction — Interview Questions
Answer aloud before selecting Show Answer for each explanation.
KEY TAKEAWAY
Compression Is Valuable Only When Important Information Survives
Define which information matters.
Learn transforms on training data.
Retain justified directions or factors.
Check variance, reconstruction and task score.
Watch original and transformed distributions.
A smaller representation is not automatically smarter; it becomes useful when it removes cost and redundancy without removing the signal the decision needs.
EXTRA TIPS
Eight Practical Reduction Habits
Split before fitting imputation, scaling, PCA, SVD or LDA.
Standardize when feature units should have comparable influence.
Inspect scree plots and cumulative variance instead of using one blind threshold.
Validate component count with the downstream model and metric.
Inspect loadings and reconstruction examples to understand lost information.
Use TruncatedSVD rather than centred PCA for large sparse matrices.
Repeat nonlinear embeddings across seeds and reasonable settings.
Version the fitted transformation together with the prediction model.
EXTRA PRACTICE QUESTIONS
Strengthen Projection and Compression Reasoning
Calculate intermediate matrices and defend every design choice.
- 01
Centre a 4×2 dataset and verify each centred column mean is zero.
- 02
Calculate a 2×2 sample covariance matrix from centred observations.
- 03
Verify that a supplied vector is an eigenvector of a covariance matrix.
- 04
Order eigenpairs and calculate every explained-variance ratio.
- 05
Project three points onto one principal component.
- 06
Reconstruct those points and calculate RMSE.
- 07
Explain how feature standardization changes PCA.
- 08
Calculate storage for an m×n matrix and its rank-K SVD factors.
- 09
Rebuild a matrix from one singular triplet.
- 10
Connect singular values to PCA covariance eigenvalues.
- 11
Explain why PCA can remove a low-variance predictive direction.
- 12
Compare PCA and LDA for a three-class labelled dataset.
- 13
List three invalid conclusions from a t-SNE plot.
- 14
Design a leakage-safe search over PCA components and classifier settings.