Welcome to the course!
How do machines learn to understand and generate human language? In this course, we build the answer from the ground up: from tokenization and statistical language models, through neural networks and the Transformer, to pretraining, fine-tuning, alignment, and the frontiers of large language models.
There is no official textbook, but we will use the following resources for reference:
Additional readings (research papers) are listed per lecture in the schedule below.
Lectures are held Tuesdays and Thursdays. No lectures during Reading Week (October 10–18). Slides will be made available before each lecture.
| Week | Tue | Thu |
|---|---|---|
| Part I: Foundations (Weeks 1–4) | ||
| 1 | No lecture: UW classes begin Wed, Sep 9 |
Lecture 1 (Thu, Sep 10)
Introduction & Fundamentals What is NLP? A brief history from Shannon to GPT. What is a language model? Probability review (chain rule, Bayes' theorem). Information theory: entropy, cross-entropy, perplexity. [Slides: PDF] |
| 2 |
Lecture 2 (Tue, Sep 15)
Words: Tokenization and Morphology What is a word? Morphological typology (isolating, agglutinative, fusional). BPE from scratch. WordPiece, Unigram LM tokenizer. Cross-lingual considerations: vocabulary sharing, low-resource scripts. [Reading: Sennrich et al., 2016] [Reading: Kudo, 2018] |
Lecture 3 (Thu, Sep 17)
Distributional Semantics and Word Embeddings Distributional hypothesis. Count-based methods (PMI, SVD). Word2Vec (skip-gram, CBOW, negative sampling). [Reading: SLP 3, Chapter 6] |
| 3 |
Lecture 4 (Tue, Sep 22)
Supervised Learning Foundations From embeddings to predictions. Feature engineering vs. learned representations. Logistic regression, loss functions, SGD. Motivation for neural approaches. |
Lecture 5 (Thu, Sep 24)
Statistical Language Models N-gram models: MLE, smoothing (Laplace, Kneser-Ney). Evaluation: perplexity, bits-per-character. Shannon's experiments. [Reading: SLP 3, Chapter 3] |
| Part II: The Core Engine (Weeks 4–7) | ||
| 4 |
Lecture 6 (Tue, Sep 29)
Neural Network Foundations MLPs, backpropagation, activations. RNNs. LSTMs/GRUs and the vanishing gradient problem. |
Lecture 7 (Thu, Oct 1)
The Transformer from Scratch Attention (Q/K/V), multi-head attention, positional encoding. Layer norm, residual connections, FFN. Full encoder-decoder walkthrough. |
| 5 |
Lecture 8 (Tue, Oct 6)
Neural Language Models RNN LMs. Transformer decoder (GPT-style). Training: teacher forcing, cross-entropy loss. Building a character-level LM from scratch. [Reading: Holtzman et al., 2020] |
Lecture 9 (Thu, Oct 8)
Masked LMs and Pretraining Objectives Autoregressive vs. masked LMs. BERT (MLM + NSP). T5 (span corruption). Comparison of pretraining objectives. [Reading: Devlin et al., 2019] [Reading: Raffel et al., 2019] |
| Reading Week (Oct 10–18): No classes | ||
| 6 | Midterm exam (Tue, Oct 20) Exam held during class time, covering Lectures 1–9 (up to reading week). Format details TBA. |
Lecture 10 (Thu, Oct 22)
Pretraining at Scale Data pipelines (web crawl, filtering, dedup, mixing). Scaling laws (Chinchilla). Optimization (AdamW, LR schedules, warmup). Mixed precision. [Reading: Radford et al., 2018] [Reading: Brown et al., 2020] |
| Part III: Understanding and Improving (Weeks 7–11) | ||
| 7 |
Lecture 11 (Tue, Oct 27)
Syntax and Context-Free Grammars Now that we've built powerful LMs, what do they learn about language structure? Constituency vs. dependency. CFGs: rules, derivations, parse trees. PCFGs: generative model, string probability, inside algorithm. [Reading: SLP 3, Chapter 18] [Reading: SLP 3, Chapter 19] |
Lecture 12 (Thu, Oct 29)
Probing Neural LMs I: Syntactic Knowledge Subject-verb agreement tests. Targeted syntactic evaluation. Structural probes. BLiMP. |
| 8 |
Lecture 13 (Tue, Nov 3)
Mechanistic Interpretability and Knowledge Probing Circuits, induction heads, superposition. Activation patching, sparse autoencoders. Factual knowledge (LAMA probes). Information-theoretic probing (MDL). [Reading: Olsson et al., 2022] |
Lecture 14 (Thu, Nov 5)
Evaluation and Benchmarking Perplexity vs. downstream evaluation. Benchmarks: MMLU, GPQA, HELM, BIG-Bench. Contamination, calibration, statistical significance. LM-as-judge, Chatbot Arena. Generation metrics: BLEU, ROUGE, BERTScore and their failure modes. [Reading: Liang et al., 2022] [Reading: Zheng et al., 2023] |
| 9 |
Lecture 15 (Tue, Nov 10)
Decoding Strategies and In-Context Learning Greedy, beam search, nucleus/top-k/temperature sampling. Repetition and degeneration. In-context learning: few-shot prompting, instruction following. Why does ICL work? Induction-head connection, Bayesian view. [Reading: Xie et al., 2022] |
Lecture 16 (Thu, Nov 12)
Supervised Fine-tuning Pretraining→finetuning paradigm. Instruction tuning: data formats, templates. Parameter-efficient methods: LoRA, adapters. [Reading: Zhou et al., 2023] |
| 10 |
Lecture 17 (Tue, Nov 17)
RLHF and Alignment Reward modeling. RLHF with PPO. DPO (direct preference optimization). Constitutional AI, RLAIF. [Reading: Ouyang et al., 2022] [Reading: Rafailov et al., 2023] |
Lecture 18 (Thu, Nov 19)
Reasoning and Test-Time Scaling Chain-of-thought, self-consistency, tree of thought. Inference-time vs. training-time compute tradeoffs. Best-of-N, verifiers, process reward models. RL from verifiable rewards: o1/R1-style reasoning models. [Reading: Wei et al., 2022] [Reading: Yao et al., 2022] [Reading: Lightman et al., 2023] [Reading: Shao et al., 2024] [Reading: Chen et al., 2026] |
| Part IV: The Frontier (Weeks 11–12) | ||
| 11 |
Lecture 19 (Tue, Nov 24)
Efficient Architectures, Long Context, and Systems Mixture of Experts (sparse routing). Linear attention, state-space models (Mamba). Quantization, distillation. Long context: RoPE, ALiBi, position interpolation. Memory hierarchy, KV cache, FlashAttention. Distributed training and inference. [Reading: Shazeer et al., 2017] [Reading: Dai et al., 2024] [Reading: Su et al., 2021] [Reading: Dao et al., 2022] |
Lecture 20 (Thu, Nov 26)
Safety, Bias, and Societal Impact Memorization and privacy; differential privacy. Hallucination and factuality. Bias, stereotypes, and fairness. Watermarking and detection of generated text. Dual-use, misuse, and environmental cost. Copyright and data provenance. [Reading: Bender et al., 2021] [Reading: Carlini et al., 2021] [Reading: Kirchenbauer et al., 2023] |
| 12 |
Lecture 21 (Tue, Dec 1)
RAG, Agents, and Open Problems Retrieval-augmented generation. LM-based agents (tool use, planning). Open problems (reasoning, grounding, multilinguality) and outlook. [Reading: Lewis et al., 2020] [Reading: Jimenez et al., 2024] [Reading: Xie et al., 2024] |
Lecture 22 (Thu, Dec 3)
Multimodality and Grounding From pixels to tokens: image patches, discrete visual tokens. Contrastive image–text alignment: CLIP; zero-shot transfer. Visual instruction tuning: LLaVA; Flamingo, BLIP. Natively multimodal and any-to-any models (Gemini, Chameleon). Multimodal generation: text-to-image/video/audio. The symbol grounding problem; visual, spatial, and embodied grounding. [Reading: Radford et al., 2021] [Reading: Liu et al., 2023] |
Assignments: 35%. Four assignments, done individually. Each assignment includes Kaggle leaderboards for quantitative evaluation and a graded analysis write-up.
Midterm exam: 25%. Held during class time on Tue, Oct 20; covers Lectures 1–9.
Final exam: 40%. Comprehensive exam covering the full course.
All assignments are due at 11:59 PM Eastern Time on the specified due date. A universal 72-hour grace period applies to all assignment deadlines:
Extensions beyond the grace period are considered only for medical emergencies: a valid Verification of Illness Form (VIF) must be submitted before the original due date, and it must cover both the original due date and the entire grace period.