Module Assessment

Overview

This assessment brings together the four lessons in module 15: text preprocessing and vectorization (tokenization, stopwords, stemming, lemmatization, CountVectorizer, TF-IDF), word embeddings and transformers (GloVe, self-attention, Hugging Face DistilBERT), and NLP workflows (text classification, named entity recognition, summarization, zero-shot classification). You will build a complete text classification pipeline from raw text to a tuned model, compare bag-of-words and neural approaches, and practice the error analysis skills needed to diagnose real-world NLP failures.

Learning Objectives

By the end of this assessment, you will have demonstrated the ability to:

• Build a complete text classification pipeline from raw text to model evaluation.
• Compare bag-of-words (TF-IDF) and transformer-based approaches.
• Preprocess text and analyze the effect of preprocessing choices on model performance.
• Interpret classification reports and analyze common misclassifications.

Starter Code

Complete the assessment using this Colaboratory notebook. Submit the link to the AI Grader for grading

The Dataset

20 Newsgroups — a collection of approximately 18,000 newsgroup posts across 20 categories. For this assessment you will use a four-category subset that represents two science topics and two non-science topics, making the multi-class boundaries meaningful and non-trivial.

from sklearn.datasets import fetch_20newsgroups

# Use 4 categories for a manageable multi-class problem
categories = ['sci.med', 'sci.space', 'rec.sport.hockey', 'talk.politics.guns']

train = fetch_20newsgroups(subset='train', categories=categories,
                           remove=('headers', 'footers', 'quotes'))
test  = fetch_20newsgroups(subset='test',  categories=categories,
                           remove=('headers', 'footers', 'quotes'))

print(f"Train: {len(train.data)} documents")
print(f"Test:  {len(test.data)} documents")
print(f"Categories: {train.target_names}")

Output:

Train: 2170 documents
Test:  1447 documents
Categories: ['rec.sport.hockey', 'sci.med', 'sci.space', 'talk.politics.guns']

Coding Assessment

Practice the concepts from this module using the included Colaboratory notebook. After completing all tasks, save your notebook to GitHub and submit the link for grading.

Task 1: Exploratory Text Analysis

Load the dataset. For each category, compute and print: the number of documents, the average document length in characters, and the 3 most common non-stopword tokens (use NLTK stopwords or a simple frequency count after lowercasing and splitting on whitespace). Write 2 sentences describing any differences you observe between categories — for example, whether one category uses longer documents or whether the vocabulary differs noticeably.

import nltk
from collections import Counter

nltk.download('stopwords')
from nltk.corpus import stopwords

stop = set(stopwords.words('english'))

for label_id, category in enumerate(train.target_names):
    docs = [train.data[i] for i in range(len(train.data))
            if train.target[i] == label_id]

    avg_len = sum(len(d) for d in docs) / len(docs)

    all_tokens = []
    for doc in docs:
        tokens = [t.lower() for t in doc.split() if t.lower() not in stop
                  and t.isalpha()]
        all_tokens.extend(tokens)

    top3 = Counter(all_tokens).most_common(3)

    print(f"\n{category}")
    print(f"  Documents: {len(docs)}")
    print(f"  Avg length (chars): {avg_len:.0f}")
    print(f"  Top tokens: {top3}")

Task 2: TF-IDF Baseline Pipeline

Build a Pipeline with TfidfVectorizer(max_features=5000, stop_words='english', ngram_range=(1,2)) followed by LogisticRegression(max_iter=1000). Train on the training split and evaluate on the test split. Print the full classification report.

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

pipe_lr = Pipeline([
    ('tfidf', TfidfVectorizer(max_features=5000,
                              stop_words='english',
                              ngram_range=(1, 2))),
    ('clf',   LogisticRegression(max_iter=1000, random_state=42)),
])

pipe_lr.fit(train.data, train.target)
y_pred = pipe_lr.predict(test.data)

print(classification_report(test.target, y_pred,
                             target_names=train.target_names))

Task 3: Preprocessing Ablation

Compare three TF-IDF configurations paired with the same LogisticRegression(max_iter=1000). Report accuracy for each configuration and identify which performs best on the test set.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

configs = {
    'A — no preprocessing':           TfidfVectorizer(),
    'B — stop_words':                 TfidfVectorizer(stop_words='english'),
    'C — stop_words + min_df=5':      TfidfVectorizer(stop_words='english',
                                                       min_df=5),
}

for name, vec in configs.items():
    pipe = Pipeline([('tfidf', vec),
                     ('clf',   LogisticRegression(max_iter=1000,
                                                  random_state=42))])
    pipe.fit(train.data, train.target)
    acc = pipe.score(test.data, test.target)
    print(f"{name}: {acc:.4f}")

Write 1–2 sentences explaining which configuration performed best and why you think that is.

Task 4: Naive Bayes Comparison

Train a Pipeline with TfidfVectorizer(stop_words='english') and MultinomialNB(). Compare accuracy and per-class F1 with the Logistic Regression from Task 2. Fill in the comparison table with values from your classification reports.

from sklearn.naive_bayes import MultinomialNB

pipe_nb = Pipeline([
    ('tfidf', TfidfVectorizer(stop_words='english')),
    ('clf',   MultinomialNB()),
])

pipe_nb.fit(train.data, train.target)
y_pred_nb = pipe_nb.predict(test.data)

print(classification_report(test.target, y_pred_nb,
                             target_names=train.target_names))

Fill in the table with the values reported by classification_report:

Task 5: Error Analysis

Using the best-performing model from Tasks 2–4, identify 3 misclassified documents from the test set. For each document, print the first 200 characters of text, the true label, and the predicted label. Write 2 sentences explaining why each document was difficult to classify correctly.

best_pipe = pipe_lr   # or whichever model performed best

y_pred_best = best_pipe.predict(test.data)
misclassified_idx = [i for i in range(len(test.data))
                     if y_pred_best[i] != test.target[i]]

for idx in misclassified_idx[:3]:
    true_label = train.target_names[test.target[idx]]
    pred_label = train.target_names[y_pred_best[idx]]
    text_snippet = test.data[idx][:200].replace('\n', ' ')

    print(f"Index: {idx}")
    print(f"  True:      {true_label}")
    print(f"  Predicted: {pred_label}")
    print(f"  Text:      {text_snippet}")
    print()

Task 6: Reflection

Answer each question in 2–4 sentences in your notebook:

1. What information does TF-IDF capture that raw term frequency (TF) alone does not? Why does this matter for distinguishing newsgroup categories?
2. In what scenario would you choose a transformer (for example, DistilBERT) over TF-IDF + Logistic Regression for a text classification task?
3. The headers, footers, and quotes were removed from the dataset using remove=('headers', 'footers', 'quotes'). Why is this important for a fair evaluation? What would happen to accuracy if you left them in?

Grading Rubric

Knowledge Check