Computer Engineering is coming soon

We're still writing and reviewing the notes for every subject, and we'll open it when every subject is ready. Meanwhile you can practise aptitude, ask Aria about any subject, or see the branches that are ready.

Feature engineering

Feature engineering means converting raw data into informative features (numeric or categorical) that help machine-learning models learn better; it covers scaling, encoding, missing-value handling, transformations, interaction and polynomial features, binning, and leakage avoidance.

Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.

Why it matters

Feature engineering is the practical step between raw data and machine-learning models. Good features speed up learning, improve final accuracy, and reduce model complexity; poor features cause bias, slow convergence and leakage. Engineers use feature engineering when preparing datasets for training, validation and deployment pipelines.

Key ideas

  • Scaling (normalisation / standardisation): methods that change numeric features to a common range or distribution to help optimisation and distance-based methods. Two common methods are min–max scaling and z-score standardisation (also called standard scaling).

    • Min–max scaling maps a feature x to x' in [a, b] (commonly [0,1]). It assumes feature values are within a finite range and sensitive to outliers.
    • Z-score standardisation maps a feature x to x' with zero mean and unit variance; it assumes the sample mean and variance are representative (requires at least two samples).
  • Encoding categorical variables: converting categories into numbers so models can use them.

    • Label encoding maps categories to integers (ordinal). It assumes an order—do not use when no order exists.
    • One-hot encoding creates a binary feature per category (nominal). It increases dimensionality; use sparse representations when many categories.
    • Target (mean) encoding replaces category by average target value; it risks data leakage and requires careful cross-validation and smoothing.
  • Missing-value imputation: replacing absent values with estimates. Simple methods: mean/median/mode imputation (assumes missing completely at random), forward/backward-fill for time series, and model-based imputation (e.g., KNN or iterative imputer). Always mark imputed values with an additional boolean feature indicating missingness to let the model learn patterns of missingness.

  • Transformations to handle skew / heteroscedasticity: apply monotonic transforms to stabilize variance and make distributions closer to Gaussian for models that assume normal errors (e.g., linear regression). Common transforms: log, square root, Box–Cox (requires positive data), and Yeo–Johnson (works with zero/negative).

  • Polynomial and interaction features: create powers and products of features to let linear models capture nonlinear relations. If original features are x1..xd, polynomial features of degree p include terms x_i^k and cross-products. This increases dimensionality combinatorially — compute the number of terms before using.

  • Binning (discretisation): convert continuous features into ordinal bins. Methods: equal-width, equal-frequency (quantile) and supervised binning (based on target). Binning reduces sensitivity to small measurement errors but discards information.

  • Outlier handling: detect outliers (IQR rule, z-score threshold, robust Mahalanobis, isolation forest) and decide to clip, transform, or remove. Be explicit about assumptions: e.g., IQR rule assumes median/IQR stable and outliers are rare.

  • Feature leakage: a feature leaks if it contains information that would not be available at prediction time (label leakage). Always design features using only information that would be available when the model is deployed.

  • Feature pipelines and reproducibility: transform training and test data using the same fitted transformers (fit on train, apply to test). Use pipelines (for example, scikit-learn Pipeline) so training transforms and model fitting are atomic and reproducible.

  • Diagrams to draw in an exam (describe exactly):

    1. For scaling: draw two histograms side-by-side labelled “Original” (x-axis: feature value, y-axis: frequency) and “After standardisation” (x-axis: z, labelled mean 0, ticks at -1,0,1; y-axis frequency). Indicate mean and std with vertical lines. Caption: "Before and after z-score standardisation".
    2. For encoding: draw a table with a column "Color" having values {red, green, blue} and show a one-hot expansion to three columns Color_red, Color_green, Color_blue with 0/1 entries.
    3. For polynomial features: draw two axes x1,x2 and show points; annotate new axes x1^2 and x1*x2 to show how linear separation in transformed space corresponds to nonlinear boundary in original space.

Formulas

x' = (x - min(x)) / (max(x) - min(x))
where x is the original scalar feature value (unit: same as original), x' is the scaled value (dimensionless), applies when performing min–max scaling to [0,1] and max(x)≠min(x). Compute min(x) and max(x) on the training set only and reuse them for validation/test/deployment to avoid leakage.

x' = a + (b - a) * (x - min(x)) / (max(x) - min(x))
where a,b are desired bounds (e.g., 0 and 1), x and x' units as above; applies when scaling to [a,b]. Fit min(x) and max(x) on the training set only; apply the same parameters to validation/test/deployment data.

z = (x - μ) / σ
where x is the original value (unit: same as x), μ is the sample mean of the feature (same unit), σ is the sample standard deviation (same unit), z is dimensionless; applies for z-score standardisation when σ>0. Compute μ and σ on the training set only and reuse them for validation/test to avoid leakage.

μ = (1/n) * Σ_{i=1..n} x_i
where μ is mean (same unit as x), n is number of samples, applies when computing sample mean.

σ = sqrt( (1/(n-1)) * Σ_{i=1..n} (x_i - μ)^2 )
where σ is sample standard deviation (same unit as x), uses Bessel's correction (n-1); applies to compute standard deviation from sample. Note: many ML libraries (e.g., scikit-learn StandardScaler) use population variance with n (not n−1) when fitting scalers, so their scaled values may differ slightly.

x' = log(x + c)
where log is natural logarithm, x must satisfy x + c > 0 (unitless after transform), c is shift constant (unit same as x); use to reduce right skew. Choose c=1 if x is integer-counts with zeros.

x_boxcox = (x^λ - 1) / λ if λ ≠ 0; x_boxcox = log(x) if λ = 0
where x>0, λ (dimensionless) is Box–Cox parameter; x_boxcox is transformed feature (dimensionless). Applies when using Box–Cox transform.

#polynomial_terms(d, p) = C(d + p, p) = (d + p)! / (d! p!)
where d is original feature count (dimensionless), p is max degree (dimensionless), result is number of terms including bias (dimensionless). Use to compute combinatorial growth of polynomial features.

IQR = Q3 - Q1
where Q1 and Q3 are first and third quartiles (same units as x); apply IQR rule for outliers: values < Q1 - 1.5·IQR or > Q3 + 1.5·IQR are often treated as outliers.

encoded_value = (Σ target_i for category k + α·global_mean) / (count_k + α)
where α is smoothing parameter (dimensionless), applies for smoothed target (mean) encoding to reduce overfitting; units same as target (e.g., probability).

Worked examples

Example 1 — Numeric scaling (min–max and z-score). Given feature values in metres: x = [0.5, 1.2, 0.9, 3.0, 2.1] (m). Steps:

  1. Compute min and max: min(x)=0.5 m, max(x)=3.0 m.
  2. Min–max to [0,1]: formula x' = (x - 0.5)/(3.0 - 0.5).
    • For 0.5: (0.5-0.5)/2.5 = 0.0
    • For 1.2: (1.2-0.5)/2.5 = 0.7/2.5 = 0.28
    • For 0.9: 0.4/2.5 = 0.16
    • For 3.0: 2.5/2.5 = 1.0
    • For 2.1: 1.6/2.5 = 0.64 Final min–max scaled vector: [0.00, 0.28, 0.16, 1.00, 0.64] (dimensionless).
  3. Compute sample mean μ and sample std σ (use n-1 for σ): μ = (0.5+1.2+0.9+3.0+2.1)/5 = 7.7/5 = 1.54 m. Variance = (1/(5-1)) * Σ(x_i - μ)^2 = (1/4)*[(0.5-1.54)^2 + ...] compute: deviations: [-1.04, -0.34, -0.64, 1.46, 0.56] m; squares: [1.0816,0.1156,0.4096,2.1316,0.3136]; sum = 4.052; variance = 4.052/4 = 1.013; σ = sqrt(1.013)=1.0065 m.
  4. Z-score each value: z = (x - 1.54)/1.0065
    • 0.5: (-1.04)/1.0065 = -1.033
    • 1.2: (-0.34)/1.0065 = -0.338
    • 0.9: (-0.64)/1.0065 = -0.636
    • 3.0: 1.46/1.0065 = 1.451
    • 2.1: 0.56/1.0065 = 0.556 Final z-scores: [-1.033, -0.338, -0.636, 1.451, 0.556] (dimensionless). Answer: min–max scaled vector [0.00, 0.28, 0.16, 1.00, 0.64], z-score vector [-1.033, -0.338, -0.636, 1.451, 0.556].

Example 2 — Categorical encoding and leak-risk. Given colours: Color = [red, blue, red, green, blue] and binary target Purchased = [1,0,1,0,0].

  1. One-hot encoding (columns Color_red, Color_blue, Color_green): Row1 red -> [1,0,0], Row2 blue -> [0,1,0], Row3 red -> [1,0,0], Row4 green -> [0,0,1], Row5 blue -> [0,1,0].
  2. Label encoding (arbitrary): map red->0, blue->1, green->2 gives vector [0,1,0,2,1]. Warning: this injects ordinal relations (1>0) which model may misuse.
  3. Target encoding (naive, leave-one-out not used): mean target per category: red: (1+1)/2=1.0, blue: (0+0)/2=0.0, green: 0/1=0.0. Replacing yields [1.0,0.0,1.0,0.0,0.0] — this leaks if computed with whole dataset; to avoid leakage compute encoding using cross-validation folds or smoothing with α. Thus explain: using full-data target encoding is label leakage and will inflate performance estimates. Answer: Prefer one-hot or properly cross-validated smoothed target encoding; never compute target encoding on full data used for evaluation.

Example 3 — Polynomial features combinatorics (theory type question). If d=3 original features and degree p=2, compute number of polynomial terms including degree 0 (bias): # = C(d+p, p) = C(5,2)=10. List them for x1,x2,x3: 1, x1, x2, x3, x1^2, x1 x2, x1 x3, x2^2, x2 x3, x3^2. Explain growth: for p=3, C(6,3)=20 terms. Answer: Be careful — creating polynomial features can explode dimensionality and requires regularisation or feature selection (see related topic Feature selection).

Common mistakes

  • Forgetting to apply the same scaling learned on train to test/deployment data.
  • Blindly using min–max scaling when outliers exist; prefer robust scaling (median and IQR) or z-score if Gaussian-like.
  • Not tracking units or interpreting transformed units; transforms (log, Box–Cox) change units and model intercept interpretation.
  • Creating polynomial features without computing combinatorial feature count and running out of memory.
  • Using target-encoding without smoothing or cross-validation; leads to strong overfit.

For your exam

Types of questions and what full-marks answers include:

  • Definitions: define feature, scaling, one-hot encoding, feature leakage. Full marks require precise, bolded definitions and an example.
  • Short notes: e.g., "Discuss methods to handle missing values" — include definitions, assumptions for each method, when to use, and an advantage/disadvantage for each.
  • Derivations: e.g., derive z-score formula and show that standardised data has mean 0 and (sample) variance 1. Full marks: show μ_z = 0, σ_z^2 = 1 algebraically and state assumption σ≠0.
  • Numericals: perform min–max and z-score on given small vectors; show all intermediate sums and final vectors with units. Full marks: correct arithmetic and units.
  • Programs: write a small Python snippet to fit a scaler and transform training and test sets using scikit-learn Pipeline; include sample input and exact output. Full marks: runnable code and explanation. Example (StandardScaler uses population variance, n):
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X_train = np.array([[0.5],[1.2],[0.9],[3.0],[2.1]])
X_test  = np.array([[0.5],[2.1]])

pipe = Pipeline([("scaler", StandardScaler())])
pipe.fit(X_train)
print(np.round(pipe.transform(X_train), 3).tolist())
print(np.round(pipe.transform(X_test), 3).tolist())

Exact output:

[[-1.155], [-0.378], [-0.711], [1.621], [0.622]]
[[-1.155], [0.622]]
  • Diagrams: draw histograms before/after and label means/std or show one-hot expansion table. Full marks: correct labels and caption explaining changes.

Quick check

  1. What transform maps data to zero mean and unit variance?
  2. Why is one-hot encoding preferred over label encoding for nominal data?
  3. Give one rule to detect univariate outliers using quartiles.

Answers: z-score (standardisation); because label encoding introduces an artificial order and one-hot preserves no order; values < Q1 - 1.5·IQR or > Q3 + 1.5·IQR are outliers.

Note for reviewer: I have kept definitions explicit and highlighted leakage warnings. I did not include scikit-learn code in Worked examples except conceptual pipelines; if you want runnable code examples for pipelines or imputation, I can add them. I may be unsure about whether to use n or n-1 in σ for your teaching preference—I've used sample standard deviation (n-1); verify that matches your university convention.

Finished this topic? Mark it so your progress, study plan and readiness keep up.

Stuck on something here?