PART 1 • FOUNDATIONS • LEVEL 05

Data Preparation, EDA & Feature Engineering

Turn imperfect observations into trustworthy model inputs. Audit quality, prevent leakage, clean systematically, discover structure and build reproducible features before training begins.

⏱️ 160–190 min🎯 Beginner → Interview Ready🧪 2 Interactive Labs💼 Industry Pipeline Skills

Start Learning →Open Pipeline Visualizer

RAW DATAagecityscore21Hyd82?hyd12024Vij95

VALIDATECLEANTRANSFORMFEATURES

MODEL-READY Xage_zhydscore_z−0.71−0.90.010.00.700.9

✓LEAKAGE SAFE

LEARNING OUTCOMES

By the End of This Level, You Can

01Audit schema, missingness, duplicates, ranges and class balance.

02Split data before learning transformations and prevent leakage.

03Handle missing values, outliers and inconsistent records responsibly.

04Encode categories, scale numbers and transform skewed features.

05Perform focused EDA and engineer useful domain-aware features.

06Build reproducible scikit-learn preprocessing pipelines.

QUICK REVISION

Six Checks Before Touching the Data

Good preprocessing begins with questions, not automatic transformations.

SHAPERows × columns

Confirm whether each row represents one valid observation.

SCHEMANames and types

Separate numeric, categorical, temporal, text and target fields.

MISSINGNESSWhere and why?

Count nulls and investigate their collection mechanism.

DISTRIBUTIONCentre, spread, skew

Inspect values before choosing summaries or transformations.

TARGETMeaning and balance

Verify labels, prevalence and evaluation consequences.

IDENTIFIERSNot ordinary features

IDs can memorize records and create misleading performance.

CONCEPT 01

Data Quality Is Multidimensional

A dataset can be complete yet still be inaccurate, inconsistent or unsuitable for the intended prediction.

COMPLETERequired values exist

Measure missingness by row, column, group and time.

VALIDValues obey rules

A percentage must be 0–100; a date must parse correctly.

CONSISTENTRepresentations agree

Hyderabad, hyd and HYD need one convention.

ACCURATEValues reflect reality

Valid-looking data can still be recorded incorrectly.

UNIQUENo unintended repetition

Duplicate entities can bias both training and evaluation.

TIMELYAvailable at prediction time

Future or delayed variables may silently leak answers.

Data contractschema + ranges + null rules + uniqueness + time rulesValidate at every pipeline entry

DETAILED EXPLANATION

Data quality includes accuracy, completeness, consistency, uniqueness, validity, timeliness and representativeness. A table can be technically complete yet still be unsuitable because labels are delayed, measurements use inconsistent units or important groups are missing. Quality must be judged against the intended prediction decision.

WORKED INTUITION

An age column without missing values is still invalid if some records contain 250 or mix years with months.

AI / PLACEMENT CONNECTION

A data-quality report becomes the first defence against unreliable model behaviour and production surprises.

COMMON MISCONCEPTION

Cleaning every unusual value can erase rare but important cases. Investigate before modifying.

CONCEPT 02 • MOST IMPORTANT

Split First: Prevent Data Leakage

Any information unavailable at real prediction time must not influence training features or learned preprocessing.

1Raw observations

Keep the untouched source as the reproducible starting point.

→

2Train / validation / test

Split by the unit and time structure of the real problem.

→

3Fit on train only

Learn medians, categories, scaling and selection from training data.

→

4Transform others

Apply the already-fitted operations to validation, test and production.

TARGET LEAKAGE

A feature directly or indirectly contains the answer.

TRAIN–TEST CONTAMINATION

Test information affects imputation, scaling, selection or tuning.

TIME LEAKAGE

Future information is used to predict an earlier outcome.

GROUP LEAKAGE

The same person, device or entity appears across splits.

Placement rule: fit preprocessing on training data; use only transform on validation and test data.

DETAILED EXPLANATION

Split the data before learning imputation values, scaling statistics, category vocabularies, feature selections or resampling decisions. Each fitted transformation must learn only from training data. Validation and test rows may pass through those fitted transformations, but they must not influence them.

WORKED INTUITION

Computing the global mean before splitting lets test values influence the replacement used for training missing values.

AI / PLACEMENT CONNECTION

A pipeline ensures cross-validation refits preprocessing inside each training fold and prevents subtle leakage.

COMMON MISCONCEPTION

Leakage can come from time, duplicate entities, post-outcome fields or preprocessing—not only from including the target column.

CONCEPT 03

Missing Values: Understand Before Imputing

A missing value can be random noise, a systematic collection pattern or useful information itself.

MCARMissing completely at random

Missingness is unrelated to observed or unobserved values. Rare in practice.

MARExplained by observed data

Missingness may depend on another recorded feature.

MNARDepends on the missing value

The unobserved value influences whether it is missing; domain reasoning is essential.

MethodUseful whenImportant caution

Drop rows/columnsLoss is small and defensibleCan introduce bias or discard signal

Mean / median / modeSimple baseline; median resists outliersReduces natural variance

Constant + indicatorMissingness itself may carry informationChoose a value distinguishable from real data

KNN / iterative imputationOther features predict missing valuesMore compute and leakage risk

DETAILED EXPLANATION

Missingness may be random, related to observed variables or related to the missing value itself. Begin by measuring patterns and understanding how values were collected. Imputation preserves rows but adds assumptions; a missing-indicator feature can preserve information about the absence itself.

WORKED INTUITION

Median income imputation is more robust than the mean when a few incomes are extremely high, but it still reduces natural variability.

AI / PLACEMENT CONNECTION

Different column types need different strategies fitted within the modelling pipeline.

COMMON MISCONCEPTION

Dropping every row with any missing value can shrink the dataset and systematically remove important groups.

CONCEPT 04

Duplicates, Types and Inconsistent Categories

Cleaning must preserve meaning while making representation reliable.

01Define the entity key

Duplicate rows and duplicate real-world entities are different problems.

02Parse deliberately

Convert dates, numeric strings and booleans with invalid-value checks.

03Standardize categories

Trim spaces, normalize case and map known aliases using a documented dictionary.

04Preserve raw columns

Keep original values or an audit log when transformations affect traceability.

df["city"] = (df["city"].str.strip().str.lower()
              .replace({"hyd": "hyderabad", "vja": "vijayawada"}))
df["joined"] = pd.to_datetime(df["joined"], errors="coerce")

DETAILED EXPLANATION

Duplicates can represent accidental repeated records, legitimate repeated events or conflicting measurements. Data types define valid operations, while category normalization joins spelling, case and whitespace variants into a controlled vocabulary. Domain rules should validate units, ranges and identifiers.

WORKED INTUITION

‘Hyderabad’, ‘hyderabad ’ and ‘HYD’ may refer to one city but require a documented mapping rather than arbitrary replacement.

AI / PLACEMENT CONNECTION

Schema and uniqueness checks can stop bad batches before they reach model training or inference.

COMMON MISCONCEPTION

Removing duplicates using all columns may miss duplicate entities, while using too few columns may delete valid repeated events.

CONCEPT 05

Outliers: Error, Rare Case or Valuable Signal?

Detection is not deletion. First determine whether an unusual value is impossible, influential or genuinely important.

Q1MEDIANQ3OUTLIER

IQR RULEBelow Q1−1.5×IQR or above Q3+1.5×IQR

Robust, distribution-free screening for numeric features.

Z-SCORE|z| > chosen threshold

Most appropriate for roughly symmetric, bell-shaped data.

DOMAIN LIMITSImpossible or implausible range

Often stronger than a generic statistical threshold.

INVESTIGATE→CORRECTORCAP / TRANSFORMORKEEP & MODEL

DETAILED EXPLANATION

An outlier is an observation far from typical values, but its meaning depends on context. It may be an error, a rare valid case, a distribution tail or evidence of a separate process. Inspect source records and model sensitivity before capping, transforming, isolating or retaining it.

WORKED INTUITION

A very large transaction may be fraud—the exact signal a fraud model needs—rather than a value to discard.

AI / PLACEMENT CONNECTION

Robust scaling, tree models and robust losses can reduce sensitivity without hiding cases.

COMMON MISCONCEPTION

Applying fixed z-score rules assumes a roughly symmetric distribution and can mislabel valid skewed values.

CONCEPT 06

Feature Scaling and Numeric Transformation

Scaling changes representation, not information. It matters most when distance, magnitude or optimization drives learning.

STANDARDIZATIONz = (x − μ) / σ

Centres near 0 with standard deviation 1. Common for linear models, SVMs, PCA and neural networks.

MIN–MAX SCALINGx′ = (x−min)/(max−min)

Maps training values to a chosen range but reacts strongly to outliers.

ROBUST SCALING(x − median) / IQR

Uses robust statistics when extreme values are present.

LOG / POWER TRANSFORMReduce right skew

Useful for positive quantities spanning several orders of magnitude.

Usually scale k-NN • K-Means • SVM • PCA • Linear/Logistic • Neural NetsUsually not required Decision Trees • Random Forest • Gradient-Boosted Trees

DETAILED EXPLANATION

Standardization centres a feature and measures it in standard-deviation units. Min–max scaling maps observed training bounds to a range. Log and power transforms can reduce positive skew. Scaling changes numerical representation, not the underlying information, and must be fitted using training statistics.

WORKED INTUITION

Distance-based and gradient-based models can be dominated by a rupee-valued feature unless scales are made comparable.

AI / PLACEMENT CONNECTION

k-NN, SVM, PCA and regularized linear models are especially sensitive to feature scale.

COMMON MISCONCEPTION

Tree splits are usually scale-invariant, so scaling them is often unnecessary rather than harmful.

CONCEPT 07

Encoding Categorical Features

The encoding must respect whether a category is nominal, ordered, high-cardinality or unseen at inference.

REDBLUE→1 00 1

One-Hot Encoding

Safe default for low-cardinality nominal categories. Avoids inventing order.

LOWMEDIUMHIGH→0 • 1 • 2

Ordinal Encoding

Use only when category order is real and meaningful.

AAB→.67 • .67 • .33

Frequency Encoding

Compact for high-cardinality fields but can hide category identity.

CATEGORYTARGET→OOF MEAN

Target Encoding

Powerful but leakage-prone; learn it out-of-fold using training data only.

Production requirement: Define how unseen categories are handled. In scikit-learn, OneHotEncoder(handle_unknown="ignore") is a common safe choice.

DETAILED EXPLANATION

One-hot encoding represents unordered categories with separate indicator columns. Ordinal encoding assigns ordered levels only when a meaningful order exists. High-cardinality categories may need grouping, hashing or carefully cross-fitted target encoding to control dimensionality and leakage.

WORKED INTUITION

Education levels may be ordinal, but assigning red=0, blue=1 and green=2 invents a false ordering for colours.

AI / PLACEMENT CONNECTION

handle_unknown settings keep inference working when new categories appear after deployment.

COMMON MISCONCEPTION

Target encoding calculated on the same rows being transformed can directly leak their labels.

CONCEPT 08

EDA: Ask Focused Questions with Evidence

Exploratory Data Analysis should reveal quality issues, distribution shape, relationships and possible leakage—not become a collection of decorative plots.

UNIVARIATEOne feature

Distribution, missingness, cardinality, centre, spread and unusual values.

BIVARIATEFeature vs feature/target

Association, separation, non-linearity and group differences.

MULTIVARIATECombined structure

Interactions, confounding, redundancy, segments and data clusters.

Is the sample representative?Do distributions change over time?Are missing values systematic?Does a feature reveal the target?Are groups treated differently?Which observations drive conclusions?

DETAILED EXPLANATION

EDA is question-driven examination of structure, quality and relationships. Use univariate summaries for distributions, bivariate views for feature–target relationships and subgroup or time views for hidden variation. Compare training and later data without using the final test target to guide model choices.

WORKED INTUITION

A feature may correlate with the target overall but reverse direction within age groups, revealing a confounding pattern.

AI / PLACEMENT CONNECTION

EDA produces hypotheses, validation rules and monitoring expectations rather than proof of causality.

COMMON MISCONCEPTION

Generating many charts without a question encourages selective storytelling and missed data problems.

CONCEPT 09

Class Imbalance and Rare Outcomes

High accuracy can hide a model that never detects the minority class.

MAJORITY950

MINORITY50

Stratified splitting

Preserve target proportions while creating evaluation sets.

Class weights

Increase the loss contribution of costly minority mistakes.

Under/oversampling

Change the training distribution without altering the test set.

SMOTE carefully

Create synthetic training examples after the split, preferably inside cross-validation.

Threshold tuning

Choose a decision threshold based on costs and operating needs.

Suitable metrics

Use precision, recall, F1, PR-AUC and confusion matrices.

DETAILED EXPLANATION

Class imbalance means one outcome is much rarer than another. Accuracy may then reward a model that ignores the rare class. Use stratified or group-aware splits, relevant precision–recall metrics, class weights, threshold tuning and carefully training-only resampling.

WORKED INTUITION

With 1% fraud, predicting ‘not fraud’ always gives 99% accuracy but detects no fraud.

AI / PLACEMENT CONNECTION

The right treatment depends on the operational cost of missed positives and false alarms.

COMMON MISCONCEPTION

Oversampling before splitting duplicates minority examples into evaluation data and produces unrealistic scores.

CONCEPT 10

Feature Engineering: Express Useful Signal

Strong features represent the problem more directly while remaining available, stable and ethical at prediction time.

DOMAINBMI, price per unit, account age

Convert raw measurements into meaningful ratios or durations.

INTERACTIONSx₁ × x₂ or grouped combinations

Represent effects that appear only when features work together.

TIMEHour, weekday, recency, lag

Respect chronology; rolling statistics must use past data only.

AGGREGATIONCount, mean, frequency, trend

Summarize event histories at the correct entity and cutoff time.

TEXTLength, tokens, TF-IDF, embeddings

Choose representation appropriate to the model and task.

MISSING INDICATORWas this value absent?

Capture informative collection patterns alongside imputation.

A feature is production-ready when it is:AvailableLeakage-safeStableInterpretableAffordableFairly usable

DETAILED EXPLANATION

Feature engineering converts raw observations into signals that better express the problem. Useful features may represent ratios, elapsed time, interactions, domain categories, rolling history or aggregated behaviour available before prediction. Each feature needs a causal timeline and production source.

WORKED INTUITION

Account age at prediction time is safer than using a future closure date that is known only after churn.

AI / PLACEMENT CONNECTION

Well-designed features often improve simple models more than replacing them with greater complexity.

COMMON MISCONCEPTION

A highly predictive feature may be a disguised identifier, outcome proxy or post-event measurement.

CONCEPT 11

Feature Selection and Multicollinearity

Remove noise and redundancy without allowing evaluation data to influence selection.

FILTERScore before modeling

Variance threshold, correlation, chi-square, ANOVA and mutual information.

WRAPPERSearch with a model

Recursive feature elimination and sequential selection; more computationally expensive.

EMBEDDEDSelection during fitting

L1 regularization and tree-based importance are learned with the model.

MULTICOLLINEARITY

Highly related predictors can make linear coefficients unstable. Inspect correlations, VIF, domain redundancy or use regularization.

VIFᵢ = 1 / (1 − Rᵢ²)

DETAILED EXPLANATION

Feature selection can improve interpretability, reduce noise and control computation. Filter methods score features independently, wrapper methods evaluate subsets, and embedded methods select during training. Correlated features may share information, making individual coefficients unstable even when predictions remain useful.

WORKED INTUITION

Lasso may retain one of several correlated features while dropping others, so selection should not be read as causal importance.

AI / PLACEMENT CONNECTION

Selection must occur inside cross-validation so validation data does not influence the chosen subset.

COMMON MISCONCEPTION

Removing features based only on pairwise correlation can discard variables that add value jointly.

CONCEPT 12

Reproducible Pipelines with ColumnTransformer

Package every learned preprocessing step with the estimator so training, validation and inference follow the same path.

numeric_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler())
])

category_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("encode", OneHotEncoder(handle_unknown="ignore"))
])

preprocess = ColumnTransformer([
    ("num", numeric_pipe, numeric_columns),
    ("cat", category_pipe, categorical_columns)
])

model = Pipeline([
    ("preprocess", preprocess),
    ("estimator", LogisticRegression())
])

01No contamination

Each cross-validation fold learns transformations from its training portion.

02Same inference path

Raw production input receives exactly the training transformations.

03Tunable together

Search preprocessing and model hyperparameters in one object.

04Easier deployment

Serialize one fitted pipeline instead of disconnected steps.

DETAILED EXPLANATION

ColumnTransformer applies type-specific preprocessing, while Pipeline connects those transformations to an estimator. During fit, each step learns from training data; during predict, the stored transformations are reused in the same order. This creates one reproducible object for validation and deployment.

WORKED INTUITION

Numeric columns can use median imputation and scaling while categorical columns use frequent-value imputation and one-hot encoding.

AI / PLACEMENT CONNECTION

Grid search can tune both preprocessing and model hyperparameters without leaking across folds.

COMMON MISCONCEPTION

Manually preprocessing the entire dataset before cross-validation defeats the protection provided by the pipeline.

CONCEPT 13

Validate Data Before and After Transformation

Silent data changes can break a model even when the program still runs.

SCHEMAExpected columns and types

Reject missing required fields and unexpected structural changes.

RANGEValid numeric and date limits

Separate hard domain rules from statistical alerts.

CATEGORYAllowed and unseen labels

Measure new-category rate instead of silently ignoring drift.

DISTRIBUTIONShift from training reference

Track missingness, quantiles, prevalence and feature drift.

OUTPUT SHAPEStable transformed columns

Confirm order, count, sparsity and absence of NaN or infinity.

LINEAGEVersion and reproduce

Record source, cutoff date, code, parameters and feature definitions.

DETAILED EXPLANATION

Validation should check raw inputs, intermediate transformations and final model matrices. Verify required columns, types, allowed ranges, uniqueness, missingness, category drift, row counts and output shapes. Assertions turn silent data corruption into visible failures.

WORKED INTUITION

After one-hot encoding, confirm that the number and order of generated columns match the fitted pipeline contract.

AI / PLACEMENT CONNECTION

Production data checks provide early warning before model metrics decline.

COMMON MISCONCEPTION

Passing a schema check does not prove that feature meaning or collection behaviour stayed unchanged.

CONCEPT 14

The CodeBhavya Model-Ready Workflow

Use this sequence for projects, interviews and production systems.

01
Frame the prediction
Define observation unit, target, prediction time and success metric.
02
Freeze raw data
Preserve a reproducible, untouched source and data dictionary.
03
Split safely
Use random, stratified, grouped or time-based splitting as required.
04
Audit training data
Inspect quality, distributions, relationships and imbalance.
05
Build preprocessing
Impute, encode, transform and scale inside a pipeline.
06
Engineer and select
Add defensible features and evaluate them inside cross-validation.
07
Validate and monitor
Check transformed outputs, document assumptions and watch drift.

DETAILED EXPLANATION

The CodeBhavya workflow starts with prediction-time availability, protects evaluation evidence, audits raw quality, builds type-aware transformations, establishes a baseline, validates the complete pipeline and records the resulting feature contract. The same contract must be monitored after deployment.

WORKED INTUITION

A model-ready package includes split logic, preprocessing pipeline, estimator, metrics, schema, feature names and known limitations.

AI / PLACEMENT CONNECTION

This end-to-end explanation is stronger in placements than listing isolated Pandas commands.

COMMON MISCONCEPTION

A clean notebook is not automatically a deployable pipeline if steps rely on hidden manual decisions.

INTERACTIVE LEARNING • CODEBHAVYA PREMIUM VISUALIZER

🎬 Raw Data to Model-Ready Features — Visual Flow

Trace a small customer dataset through a leakage-safe preprocessing pipeline.

LIVE

STEP 1 OF 7

Start with raw observations

The source contains missing values, inconsistent categories and an impossible score.

Step 1 of 7

PROGRAM TRACING • TRUE LOOP FLOW

Trace Missing-Value and Range Cleaning

Watch the cursor return through every list value, impute one missing score and reject an impossible score.

CORE CLEANING PATTERN

Make Every Decision Explicit

The program imputes a documented fallback and retains only valid scores.

raw_scores = [82, None, 95, 120, 76]
clean_scores = []
for score in raw_scores:
    if score is None:
        score = 80
    if 0 <= score <= 100:
        clean_scores.append(score)
print(clean_scores)

TRACE RESULT

[82, 80, 95, 76]

82: valid and retained.
None: imputed as 80, then retained.
95: valid and retained.
120: fails the domain range and is skipped.
76: valid and retained.

PROGRAMMING PROBLEMS • CODEBHAVYA PATTERN

Build Reliable Preprocessing Skills

Attempt each problem independently. The checker rewards the requested transformation and leakage-safe structure.

0 / 5Solved independently0 / 500Best score

QUICK KNOWLEDGE CHECK

Test Preprocessing and EDA Decisions

Select one answer for every question. Results show your answer, the correct answer and an explanation.

Not checked yet

PLACEMENT CHECKPOINT

How Data Preparation Appears in Hiring Rounds

Interviewers expect the order of operations, leakage risks and trade-offs—not a list of library functions.

ROUND 01

Dataset Diagnosis

Identify missingness, duplicates, invalid values, skew and imbalance.

ROUND 02

Pipeline Design

Choose split strategy, imputation, encoding and scaling for a scenario.

ROUND 03

Leakage Detection

Find target, temporal, group and train–test contamination.

ROUND 04

Production Readiness

Handle unseen categories, schema changes, drift and reproducibility.

CodeBhavya interview pattern:Define the observation → Split safely → Fit on train → Transform consistently → Validate assumptions → Monitor change.

INTERVIEW PREPARATION

🎤 Data Preparation, EDA & Features — Interview Questions

Answer aloud before selecting Show Answer for each explanation.

KEY TAKEAWAY

A Trustworthy Data Pipeline in One View

1Define

Fix the observation, target and prediction time.

→

2Split

Protect evaluation evidence before learning anything.

→

3Prepare

Clean, impute, encode, transform and engineer.

→

4Validate

Check assumptions, outputs, lineage and drift.

A sophisticated model cannot repair information that was leaked, misunderstood or prepared inconsistently.

EXTRA TIPS

Habits of Reliable ML Practitioners

Preserve raw data and make every cleaning decision reproducible.

Split before fitting imputation, scaling, encoding or feature selection.

Use domain rules to distinguish impossible values from rare valid cases.

Compare every transformation against a simple baseline.

Inspect subgroup and time-based distributions, not only global averages.

Put preprocessing and the estimator inside one deployable pipeline.

EXTRA PRACTICE QUESTIONS

Strengthen Data Preparation Decisions

Explain the reasoning before writing code.

01
Design a data contract for age, city, joining date and placement status.
02
Explain why filling missing values before splitting can cause leakage.
03
Choose mean or median imputation for a highly skewed salary column.
04
Distinguish an exact duplicate row from a repeated customer entity.
05
Decide whether a transaction of ₹5,00,000 is an error or valuable signal.
06
Compare StandardScaler, MinMaxScaler and RobustScaler for outlier-heavy data.
07
Encode city, education level and a million product IDs appropriately.
08
List five EDA checks for a binary fraud target with 1% positives.
09
Create three leakage-safe features from a customer event history.
10
Explain why SMOTE must not be applied before train–test splitting.
11
Design a ColumnTransformer for numeric, categorical and date features.
12
List the validations required before sending transformed data to a model.