Summary

Overview

You've completed the full classification pipeline for the mushroom identification problem — from a raw CSV of single-letter codes to a decision tree that perfectly classifies every mushroom in the held-out test set. In this final lesson, you'll consolidate what the case study demonstrated, reflect on the key decisions made along the way, and think critically about what "good performance" actually means in a safety-critical context.

Learning Objectives

By the end of this lesson, you will have learned how to:

• Summarize and communicate the findings of a classification case study.
• Reflect on the practical implications of misclassification in safety-critical domains.

Starter Code

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# All variables from previous lessons should be in memory:
# df, df_clean, X, y
# X_train_enc, X_test_enc, y_train, y_test
# lr, dt, rf  (trained models)

What Was Built

The raw dataset went through a complete transformation in four steps:

Raw CSV (8,124 rows × 23 columns — all categorical strings)
    ↓ EDA: identify zero-variance feature, missing value encoding,
           and odor as near-perfect predictor
    ↓ Cleaning: drop veil_type, replace '?' with 'm' in stalk_root
    ↓ Feature Engineering: label encode 21 features, train/test split
    ↓ Model Selection: train 3 classifiers, compare, tune threshold
Final model: Decision Tree, max_depth=6 → 100% accuracy, 100% poisonous recall

Pipeline Decisions Summary

Key Insights from EDA

1. Odor Is Almost Sufficient Alone

odor_accuracy = (
    df["odor"].isin(["a", "l"])   |   # always edible
    df["odor"].isin(["c", "y", "f", "m", "p", "s"])  # always poisonous
).mean()

print(f"Fraction of mushrooms correctly classified by odor rule alone: {odor_accuracy:.1%}")

Output:

Fraction of mushrooms correctly classified by odor rule alone: 95.7%

A simple if-then rule based on odor gets 95.7% of cases right with no model at all. The remaining 4.3% — mushrooms with no odor — require additional features to classify.

2. The Dataset Is Cleanly Separable

Unlike most real-world classification problems, this dataset has a near-perfect solution. The features collectively determine class with certainty:

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score

# Check if a depth-1 tree is already sufficient
dt1 = DecisionTreeClassifier(max_depth=1, random_state=42)
print(f"Depth-1 tree CV accuracy: {cross_val_score(dt1, X_train_enc, y_train, cv=5, scoring='accuracy').mean():.3f}")

dt6 = DecisionTreeClassifier(max_depth=6, random_state=42)
print(f"Depth-6 tree CV accuracy: {cross_val_score(dt6, X_train_enc, y_train, cv=5, scoring='accuracy').mean():.3f}")

Output:

Depth-1 tree CV accuracy: 0.985
Depth-6 tree CV accuracy: 1.000

A depth-1 tree (a single split on odor) achieves 98.5% accuracy. Depth 6 handles the remaining ambiguity perfectly.

3. Feature Alignment Across Models

All three models rank these features in the top 5:

• odor
• spore_print_color
• gill_size
• gill_color

When multiple models independently identify the same features as important, it's strong evidence the signal is real — not an artifact of one algorithm's assumptions.

The Decision Tree's Rules

One of the decision tree's greatest advantages is that its logic is fully inspectable. The first three levels of the tree correspond to rules like:

If odor is in {foul, fishy, creosote, musty, pungent, spicy}:
    → Poisonous
Else if odor is in {almond, anise}:
    → Edible
Else (odor = none):
    If spore_print_color is green or buff:
        → Poisonous
    Else if gill_size is narrow:
        → Poisonous (with high confidence)
    ...

These rules are directly inspectable, explainable to a user, and auditable by a domain expert. A mycologist could review the tree and confirm whether the rules match their knowledge.

Reflecting on Safety-Critical Classification

The mushroom dataset provides a useful lens for thinking about the relationship between model accuracy and real-world consequences.

Scenario: A Foraging Application

Imagine deploying this model in a mobile app that advises hikers about mushroom safety. Consider three users:

User A encounters 100 mushrooms:

• The decision tree correctly classifies all 100
• No harm done

User B has a phone running logistic regression at the default threshold of 0.5:

• ~3 out of 100 poisonous mushrooms are predicted edible
• Depending on the specific toxins, consuming even a small amount could cause severe liver damage or death

User C has the logistic regression tuned to threshold 0.1:

• 0 poisonous mushrooms are missed
• ~13 edible mushrooms are flagged as potentially dangerous
• User avoids those mushrooms unnecessarily — a minor inconvenience

The cost asymmetry is overwhelming:

• False negative (missed poison): life-threatening
• False positive (unnecessary avoidance of edible mushroom): harmless

This is why recall on the poisonous class was the primary metric from the start, and why threshold tuning toward lower values makes sense even when it costs precision.

Generalization Caveat

The UCI mushroom dataset represents a controlled, clean benchmark. Real-world mushroom identification is harder:

• This dataset covers only 23 species of gilled mushrooms from North American temperate forests
• Specimens may be immature, damaged, or atypical
• Odor assessment is subjective and varies between observers
• Field photos introduce noise that single-letter codes don't capture

A perfect-accuracy model on this dataset does not imply that a real-world mushroom identification system would be safe to deploy without extensive validation on new species, new geographies, and real collector data.

Final Dataset Snapshot

print("=== Cleaned Dataset ===")
print(f"Rows:              {df_clean.shape[0]}")
print(f"Feature columns:   {df_clean.shape[1] - 1}   (after dropping veil_type)")
print(f"Missing values:    0")

print("\n=== Feature Matrix ===")
print(f"Train:  {X_train_enc.shape}")
print(f"Test:   {X_test_enc.shape}")

print("\n=== Best Model: Decision Tree (max_depth=6) ===")
from sklearn.metrics import accuracy_score
print(f"Accuracy:          {accuracy_score(y_test, dt.predict(X_test_enc)):.1%}")
print(f"Poisonous Recall:  {recall_score(y_test, dt.predict(X_test_enc)):.1%}")
print(f"AUC:               {roc_auc_score(y_test, dt.predict_proba(X_test_enc)[:, 1]):.4f}")

Output:

=== Cleaned Dataset ===
Rows:              8124
Feature columns:   21   (after dropping veil_type)
Missing values:    0

=== Feature Matrix ===
Train:  (6499, 21)
Test:   (1625, 21)

=== Best Model: Decision Tree (max_depth=6) ===
Accuracy:          100.0%
Poisonous Recall:  100.0%
AUC:               1.0000

What This Case Study Demonstrated

This case study is intentionally simple on the modeling side — the data is cleanly separable and the features are already well-chosen. The real challenge was on the framing side: understanding that accuracy is not enough, and that the definition of a "good" model depends entirely on what a wrong prediction costs.

Where to Go From Here

Reflection Questions

1. The ? values in stalk_root were treated as their own category m rather than dropped or imputed. Looking back at the EDA, what evidence supported this decision? What would have been lost if you had dropped those 2,480 rows?

2. The decision tree achieves 100% accuracy on this test set. Does that mean it will always achieve 100% accuracy on new mushroom data? What assumptions does this performance estimate rely on?

3. Logistic regression at threshold=0.5 achieves 97% recall on the poisonous class. Logistic regression at threshold=0.1 achieves 100% recall. If you were designing a real foraging app, which would you choose? What additional safeguards would you add beyond the model itself?