Introduction to Feature Engineering

Overview

In the previous lesson, you cleaned the raw dataset by removing errors, imputing missing values, and standardizing formatting. Now you'll take the clean data one step further: feature engineering — transforming and creating variables so that machine learning algorithms can actually use them. Most ML algorithms require purely numeric input, and even among numeric features, the scale and distribution of values affects model performance. In this lesson, you'll learn how to encode categoricals, scale numeric features, create new derived features, and understand why feature quality drives model quality.

Learning Objective

By the end of this lesson, you will have learned how to:

• Create and transform features, including encoding categorical variables.
• Recognize the impact of feature quality on model performance.

Key terms

Feature engineering: The process of transforming raw variables into representations that better capture the underlying signal for a machine learning model.

Encoding: Converting categorical (text) values into numeric representations that algorithms can process.

Label encoding: Assigning each unique category an integer (e.g., male → 0, female → 1). Suitable for ordinal categories or tree-based models.

One-hot encoding: Creating a new binary column for each unique category. Suitable for nominal (unordered) categories.

OrdinalEncoder: A scikit-learn class for label encoding.

OneHotEncoder: A scikit-learn class for one-hot encoding.

Feature scaling: Transforming numeric features to a common scale so that large-valued columns don't dominate distance-based or gradient-based algorithms.

StandardScaler: Scales features to have mean = 0 and standard deviation = 1 (z-score normalization).

MinMaxScaler: Scales features to a fixed range, typically [0, 1].

Feature creation: Deriving new columns from existing ones — for example, combining sibsp and parch into a family_size variable.

ColumnTransformer: A scikit-learn utility that applies different preprocessing steps to different columns simultaneously.

Starter Code

This lesson encourages experimentation. Use the included Colaboratory notebook to run the code as you learn, or read the lesson and then freely experiment with the notebook's code.

Introduction

Even after cleaning, a DataFrame isn't ready for most ML algorithms. Two types of problems remain:

1. Categorical columns contain strings — algorithms need numbers.
2. Numeric columns may be on very different scales — a fare of $512 and an age of 22 are both valid, but many algorithms treat features with larger numeric values as more important unless they're rescaled.

Feature engineering addresses both issues, and goes further: skilled feature engineers create entirely new variables that capture relationships the raw data doesn't surface directly. The difference between a mediocre model and a great one is often not the algorithm — it's the features fed into it.

Setup: Cleaned Titanic Data

import pandas as pd
import numpy as np
import seaborn as sns
from sklearn.model_selection import train_test_split

# Load and minimally clean
df = sns.load_dataset("titanic")
df = df[["survived", "pclass", "sex", "age", "sibsp", "parch", "fare", "embarked"]]
df["age"]      = df["age"].fillna(df["age"].median())
df["embarked"] = df["embarked"].fillna(df["embarked"].mode()[0])
df["fare"]     = df["fare"].clip(lower=0.01)
df = df.dropna()

# Split before any feature engineering
X = df.drop(columns="survived")
y = df["survived"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Train: {X_train.shape}, Test: {X_test.shape}")

Encoding Categorical Variables

Why Encoding Is Necessary

Most algorithms perform arithmetic on inputs (distances, dot products, gradients). The string "female" cannot be added or multiplied — it must become a number.

Label Encoding

Label encoding assigns each unique value an integer. It's fast and compact, but it implies an order — which is only appropriate when order actually exists:

from sklearn.preprocessing import OrdinalEncoder

# 'pclass' is already numeric (1, 2, 3 — ordinal)
# For 'sex' (binary): male/female → 0/1 is fine
encoder = OrdinalEncoder()
X_train_encoded = X_train.copy()
X_train_encoded["sex_encoded"] = encoder.fit_transform(X_train[["sex"]])
print(X_train_encoded[["sex", "sex_encoded"]].head())

Output:

       sex  sex_encoded
0   female          0.0
1     male          1.0
...

Warning: Don't label-encode nominal categories with more than two values. Encoding embarked as C=0, Q=1, S=2 falsely implies S > Q > C — that Southampton is "more embarked" than Cherbourg, which is meaningless.

One-Hot Encoding

One-hot encoding creates a new binary column for each category. No false ordering is implied:

# With pandas
embarked_dummies = pd.get_dummies(X_train["embarked"], prefix="embarked", drop_first=True)
print(embarked_dummies.head())

Output:

   embarked_Q  embarked_S
0           0           1
1           0           0
2           0           1
...

drop_first=True drops one column to avoid multicollinearity (if you know Q=0 and S=0, you know it must be C=1, so the third column is redundant).

# With scikit-learn (pipeline-friendly)
from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(drop="first", sparse_output=False)
embarked_encoded = ohe.fit_transform(X_train[["embarked"]])
print(ohe.get_feature_names_out())

When to Use Which

Feature Scaling

Why Scaling Matters

Consider a nearest-neighbor algorithm computing the distance between two passengers:

• Passenger A: age=25, fare=512
• Passenger B: age=30, fare=10

The Euclidean distance is dominated by the fare difference (502) and barely affected by the age difference (5). This means the algorithm treats fare as ~100× more important than age — purely because of scale. Scaling removes this artifact.

Algorithms that require scaling:

• k-Nearest Neighbors (KNN)
• Support Vector Machines (SVM)
• Linear and Logistic Regression (speeds up convergence)
• Neural Networks
• PCA and other dimensionality reduction

Algorithms that do NOT require scaling:

• Decision Trees
• Random Forests
• Gradient Boosting (XGBoost, LightGBM)
• These split on individual feature values and are scale-invariant.

StandardScaler (Z-score Normalization)

Transforms each feature to have mean = 0 and standard deviation = 1:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
numeric_cols = ["age", "fare", "sibsp", "parch"]

X_train_scaled = X_train.copy()
X_train_scaled[numeric_cols] = scaler.fit_transform(X_train[numeric_cols])

print(X_train_scaled[numeric_cols].describe().round(2))

Output (mean ≈ 0, std ≈ 1):

         age   fare  sibsp  parch
mean    0.00   0.00   0.00   0.00
std     1.00   1.00   1.00   1.00

Fit only on training data. Apply the same fitted scaler to the test set:

X_test_scaled = X_test.copy()
X_test_scaled[numeric_cols] = scaler.transform(X_test[numeric_cols])

MinMaxScaler

Scales each feature to the range [0, 1]:

from sklearn.preprocessing import MinMaxScaler

mm_scaler = MinMaxScaler()
X_train_mm = mm_scaler.fit_transform(X_train[numeric_cols])

Use MinMaxScaler when you need values in a specific range (e.g., neural network inputs). Use StandardScaler as the default for most other cases.

Creating New Features

New features derived from existing ones can capture patterns the raw variables don't express directly. This is where domain knowledge is most valuable.

Combining Columns

X_train_fe = X_train.copy()

# Total family members on board
X_train_fe["family_size"] = X_train_fe["sibsp"] + X_train_fe["parch"] + 1

# Is the passenger traveling alone?
X_train_fe["is_alone"] = (X_train_fe["family_size"] == 1).astype(int)

print(X_train_fe[["sibsp", "parch", "family_size", "is_alone"]].head())

Binning Continuous Variables

# Age groups
X_train_fe["age_group"] = pd.cut(
    X_train_fe["age"],
    bins=[0, 12, 18, 35, 60, 100],
    labels=["child", "teen", "young_adult", "adult", "senior"]
)

print(X_train_fe["age_group"].value_counts())

Log Transformation for Skewed Features

import numpy as np

# fare is right-skewed — log transform makes it more symmetric
X_train_fe["log_fare"] = np.log1p(X_train_fe["fare"])

Interaction Features

# Gender × class interaction — historically, women in 1st class had very high survival
X_train_fe["sex_pclass"] = (
    X_train_fe["sex"].map({"female": 0, "male": 1}) * X_train_fe["pclass"]
)

Putting It All Together: ColumnTransformer

In a real ML pipeline, you apply different transformations to different columns simultaneously. ColumnTransformer organizes this cleanly:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

numeric_features     = ["age", "fare", "sibsp", "parch"]
categorical_features = ["sex", "embarked"]

numeric_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler",  StandardScaler())
])

categorical_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("onehot",  OneHotEncoder(drop="first", sparse_output=False))
])

preprocessor = ColumnTransformer(transformers=[
    ("num",  numeric_transformer,     numeric_features),
    ("cat",  categorical_transformer, categorical_features)
])

# Fit on training data, transform both sets
X_train_processed = preprocessor.fit_transform(X_train)
X_test_processed  = preprocessor.transform(X_test)

print(f"Processed train shape: {X_train_processed.shape}")

This pipeline guarantees that:

• The same imputation values are used for train and test
• The same scaler parameters are applied to both
• The same one-hot categories are encoded consistently

A fitted preprocessor can be saved and reused in production with joblib.dump().

Why Feature Quality Drives Model Quality

To make this concrete, consider two versions of the same Titanic dataset:

The algorithm doesn't change — only the features do. This is why experienced data scientists spend more time on feature engineering than on model selection.

Conclusion

In this lesson, you learned how to encode categorical variables using label encoding and one-hot encoding, scale numeric features with StandardScaler and MinMaxScaler, create new features through combination, binning, log transformation, and interactions, and assemble all of these steps cleanly using scikit-learn's Pipeline and ColumnTransformer. Feature engineering is one of the highest-leverage activities in ML — the same algorithm performs dramatically differently depending on the quality of its inputs. In the next lesson, you'll apply everything from this module in the module assessment.

Practice

Coding Assessment

Practice the concepts from this lesson using this notebook. After completing the exercises, save your notebook to GitHub and submit the link for grading.

Knowledge Check