Multi-label Classification

Overview

In the previous lesson, you built binary classifiers that predict one of two possible outcomes and learned how to evaluate them rigorously. Both binary and multi-class classification assume that each sample belongs to exactly one category. In this lesson, you'll explore a structurally different problem: multi-label classification, where a single sample can simultaneously belong to multiple categories. This comes up constantly in real-world applications and requires a different modeling approach.

Learning Objective

By the end of this lesson, you will have learned how to:

• Recognize problems where samples can carry multiple labels simultaneously.
• Implement multi-label classification using scikit-learn's MultiOutputClassifier.
• Evaluate multi-label models using Hamming loss and label-level metrics.

Key Terms

Multi-label classification: A classification task where each sample can be assigned zero, one, or multiple labels from a predefined set simultaneously.

Label set: The complete collection of possible labels. In a movie genre problem, the label set might be {action, comedy, drama, horror, romance, thriller}.

Binary relevance: The simplest multi-label approach — train one independent binary classifier per label and combine their outputs.

MultiOutputClassifier: A scikit-learn wrapper that applies a binary classifier independently to each output column, implementing binary relevance.

Hamming loss: The fraction of label-sample pairs that are incorrectly predicted. Lower is better. A Hamming loss of 0.1 means 10% of individual label predictions are wrong.

Label-level precision/recall: Per-label precision and recall scores, averaged across labels using macro, micro, or weighted averaging.

Jaccard similarity (subset accuracy): The fraction of samples where the predicted label set exactly matches the true label set. Very strict — any single wrong label counts as a full failure.

Classifier chain: An approach that trains classifiers sequentially, where each successive classifier uses the previous classifiers' predictions as additional features.

Starter Code

This lesson encourages experimentation. Use the included Colaboratory notebook to run the code as you learn, or read the lesson and then freely experiment with the notebook's code.

Introduction

Consider a movie on a streaming platform. It doesn't fit into just one genre:

• The Dark Knight is action, crime, and thriller
• Toy Story is animation, adventure, and comedy
• Parasite is drama, thriller, and dark comedy

Forcing it into a single category loses information. The same is true for:

• Document tagging: A news article about climate legislation is tagged "environment", "politics", and "policy"
• Medical diagnosis: A patient's chart may indicate hypertension, diabetes, and sleep apnea simultaneously
• Image labeling: A photo can contain a dog, a park, and a child at the same time
• Music categorization: A song can be "jazz", "instrumental", and "relaxing"

Multi-label classification preserves this richness. Instead of predicting a single class, the model outputs a binary vector — one entry per possible label, where 1 means "this label applies" and 0 means it doesn't.

How the Target Variable Changes

In binary and multi-class classification, y is a 1D array of class labels:

# Binary / multi-class
y = [0, 1, 0, 1, 2, 1, ...]   # one integer per sample

In multi-label classification, y is a 2D matrix — one row per sample, one column per label:

# Multi-label: [action, comedy, drama, thriller, romance]
y = [
    [1, 0, 0, 1, 0],   # sample 0: action + thriller
    [0, 1, 1, 0, 0],   # sample 1: comedy + drama
    [0, 0, 1, 1, 0],   # sample 2: drama + thriller
    [0, 0, 0, 0, 1],   # sample 3: romance only
]

This matrix format is what scikit-learn expects for multi-label tasks.

Building a Multi-label Dataset

Let's create a synthetic movie dataset to make this concrete:

import numpy as np
import pandas as pd
from sklearn.datasets import make_multilabel_classification
from sklearn.model_selection import train_test_split

# Generate a synthetic multi-label dataset
X, y = make_multilabel_classification(
    n_samples=1000,
    n_features=20,
    n_classes=5,       # 5 possible labels
    n_labels=2,        # average of 2 labels per sample
    random_state=42
)

label_names = ["action", "comedy", "drama", "thriller", "romance"]
y_df = pd.DataFrame(y, columns=label_names)

print("Label distribution:")
print(y_df.mean().round(2))

print(f"\nSample label vectors:")
print(y_df.head())

Output:

Label distribution:
action      0.44
comedy      0.39
drama       0.52
thriller    0.41
romance     0.36

Sample label vectors:
   action  comedy  drama  thriller  romance
0       1       1      1         0        0
1       0       0      1         1        0
2       0       0      0         1        1
3       1       0      1         0        0
4       1       1      0         0        0

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

Approach 1: Binary Relevance with MultiOutputClassifier

The simplest and most common approach is binary relevance: train one independent binary classifier for each label. If you have 5 labels, you train 5 separate binary classifiers.

from sklearn.multioutput import MultiOutputClassifier
from sklearn.linear_model import LogisticRegression

# Wrap a binary classifier to handle multiple outputs
base_clf = LogisticRegression(random_state=42)
multi_clf = MultiOutputClassifier(base_clf)

multi_clf.fit(X_train, y_train)
y_pred = multi_clf.predict(X_test)

print(f"Predictions shape: {y_pred.shape}")  # (200, 5)
print("First 5 predictions:")
print(pd.DataFrame(y_pred[:5], columns=label_names))

Output:

Predictions shape: (200, 5)
First 5 predictions:
   action  comedy  drama  thriller  romance
0       1       1      1         0        0
1       0       0      1         1        0
2       0       0      0         0        1
3       1       0      1         0        0
4       1       1      0         0        0

The wrapper trains 5 logistic regression models — one for each genre — and combines their predictions into a single output matrix.

Advantage: Simple, interpretable, parallelizable.

Limitation: The independent classifiers don't know about label correlations. In reality, "action" and "thriller" often co-occur, but binary relevance treats each label as isolated.

Approach 2: Classifier Chains

Classifier chains improve on binary relevance by propagating predictions from earlier classifiers as features for later ones:

• Classifier 1 predicts "action" using original features
• Classifier 2 predicts "comedy" using original features + predicted "action"
• Classifier 3 predicts "drama" using original features + predicted "action" + predicted "comedy"
• …and so on

from sklearn.multioutput import ClassifierChain

chain_clf = ClassifierChain(LogisticRegression(random_state=42), random_state=42)
chain_clf.fit(X_train, y_train)
y_pred_chain = chain_clf.predict(X_test)

The chain allows later classifiers to leverage label correlations — if "action" is predicted, the probability of "thriller" may increase. The trade-off is that early prediction errors propagate through the chain.

Evaluating Multi-label Models

Because each prediction is now a vector, standard binary metrics need adaptation.

Hamming Loss

The most common multi-label metric. It measures the fraction of label-sample pairs that are wrong:

from sklearn.metrics import hamming_loss

hl_br    = hamming_loss(y_test, y_pred)
hl_chain = hamming_loss(y_test, y_pred_chain)

print(f"Hamming Loss (Binary Relevance): {hl_br:.3f}")
print(f"Hamming Loss (Classifier Chain): {hl_chain:.3f}")

Output:

Hamming Loss (Binary Relevance): 0.213
Hamming Loss (Classifier Chain): 0.198

A Hamming loss of 0.213 means 21.3% of individual label predictions are incorrect. Lower is better; 0 is perfect.

Exact Match (Subset Accuracy)

The strictest metric — what fraction of samples have every label exactly correct?

from sklearn.metrics import accuracy_score

exact_match = accuracy_score(y_test, y_pred)
print(f"Exact match (subset accuracy): {exact_match:.1%}")

Output:

Exact match (subset accuracy): 37.0%

This is strict. Getting 4 out of 5 labels right still counts as a complete failure. Useful when partial predictions are unacceptable, but overly harsh for most applications.

Label-level Classification Report

You can compute precision, recall, and F1 for each label independently:

from sklearn.metrics import classification_report

print(classification_report(
    y_test,
    y_pred,
    target_names=label_names,
    zero_division=0
))

Output:

              precision    recall  f1-score   support

      action       0.73      0.71      0.72        84
      comedy       0.71      0.70      0.71        76
       drama       0.74      0.75      0.74       102
    thriller       0.75      0.72      0.73        80
     romance       0.72      0.69      0.70        73

   micro avg       0.73      0.72      0.72       415
   macro avg       0.73      0.71      0.72       415
weighted avg       0.73      0.72      0.72       415

This lets you identify which labels your model handles well or poorly.

Metric Summary

Which to use?

• Hamming loss for a quick overall health check
• Per-label metrics to identify weak spots
• Exact match only when partial correctness is unacceptable

When to Use Multi-label vs. Multi-class

The key question: can a single sample belong to more than one category at the same time? If yes, you need multi-label classification.

Conclusion

In this lesson, you learned that multi-label classification addresses problems where a single sample can have multiple simultaneous labels — common in document tagging, image labeling, and medical diagnosis. You implemented the binary relevance approach using MultiOutputClassifier, which trains independent classifiers per label, and saw how classifier chains improve on this by modeling label correlations. You evaluated models with Hamming loss, exact match, and per-label F1 scores. In the next lesson, you'll study multi-class classification — where you have three or more mutually exclusive categories and need strategies like One-vs-Rest to extend binary classifiers.

Practice

Knowledge Check