CS 886: Recent Advances on Foundation Models

Winter 2024

People

Instructor:
- Dr. Wenhu Chen, (wenhuchen [at] uwaterloo [dot] ca)
TAs:
- Cong Wei (cong.wei [at] uwaterloo [dot] ca)

Timetable

Lectures will take place once per week as follows

Davis Center 2585 every Wednesday from 12:00 PM - 2:50 PM
Session 1: 12:00PM - 1:20PM on Wednesday
Session 2: 1:30PM - 2:50PM on Wednesday
The Jan 10th will be instructed by me. The following lectures will be student presentation and project

Deliverables

[1] In-class Presentation. Two students will pair up to (1) create slides, (2) co-present it to the class, (3) leading the discussion period. The presentation is 80 minutes.
[2] Reading Notes. Student needs to submit one reading note regarding the recent papers of their interest. The paper could be anything in the area of deep learning and foundation models.
[3] Course Project. The group size is up to 2. The student needs to submit a high-quality 8-page report plus a final in-class presentation.
Submit presentation material & reading notes & project to LEARN .

Assessment

[1] In-class Presentation: 40%. [First submission deadline on Feb 7th. The second deadline on April 1st.]
[2] Reading Notes: 10%. [deadline on April 1st.]
[3] Course project: 50%. [1-page proposal deadline on Feb 28st. 8-page final report deadline on April 10th.]

Communication

All communication should take place using the Piazza discussion board.
We do not upload materials or assignments to Piazza, instead, these materials will appear on LEARN
Sign up for Piazza (if you're not already) here.
Public Piazza posts (can be anonymous) are the preferred method for questions about course material, etc. Students can then help each other and instructors can read/reply.
Private Piazza posts (to instructors only) can be used for any posts that contain solution snippets or private questions.
Only in exceptional cases where you need to contact only the instructor should you use the personal email above.

Project

The project needs to be related to foundation models like text or vision or multimodal models.
The project needs to contain the following sections: introduction, problem definition, algorithm design, experiments, evaluation, conclusion.
Please use Latex template as a template to write the final report.
Please make sure the code are not copied from public repository. Any violation will be seen as plagiarism with serious consequences.

Lectures

Lecture Number	Topic	Date	Slides	References
Introduction to Foundation Models
1	Foundation Models and its Applications	Jan 10th	Slides	On the Opportunities and Risks of Foundation Models Multimodal Foundation Models: From Specialists to General-Purpose Assistants Large Multimodal Models: Notes on CVPR 2023 Tutorial Towards Generalist Biomedical AI A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT Interactive Natural Language Processing Towards Reasoning in Large Language Models: A Survey
2	Course Logistics	Jan 10th		Building Teams Slides Requirements Project Requirements
3	RNN & CNN	Jan 17th	Slides Video	Recurrent Neural Networks (RNNs): A gentle Introduction and Overview Highway Networks Long Short-Term Memory Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Effective Approaches to Attention-based Neural Machine Translation An Introduction to Convolutional Neural Networks ImageNet Classification with Deep Convolutional Neural Networks U-Net: Convolutional Networks for Biomedical Image Segmentation Deep Residual Learning for Image Recognition Densely Connected Convolutional Networks Aggregated Residual Transformations for Deep Neural Networks A ConvNet for the 2020s
4	NLP & CV	Jan 17th	Slides	Sequence to Sequence Learning with Neural Networks Thumbs up? Sentiment Classification using Machine Learning Techniques A survey of named entity recognition and classification Teaching Machines to Read and Comprehend Deep neural networks for acoustic modeling in speech recognition A Neural Attention Model for Sentence Summarization Microsoft COCO: Common Objects in Context Rich feature hierarchies for accurate object detection and semantic segmentation Fully Convolutional Networks for Semantic Segmentation DeepFace: Closing the Gap to Human-Level Performance in Face Verification DeepPose: Human Pose Estimation via Deep Neural Networks Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Transformer Architecture
5	Self Attention & Transformers	Jan 24th	Slides Video	Neural Machine Translation by Jointly Learning to Align and Translate Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Attention Is All You Need Annotated Transformer Image Transformer An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
6	Efficient Transformers	Jan 24th	Slides Video1 Video2	Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Perceiver: General Perception with Iterative Attention Random Feature Attention Longformer: The Long-Document Transformer Generating Long Sequences with Sparse Transformers Linformer: Self-Attention with Linear Complexity Efficiently Modeling Long Sequences with Structured State Spaces
7	Parameter-efficient Tuning	Jan 31st	Slides Video	Parameter-Efficient Transfer Learning for NLP LoRA: Low-Rank Adaptation of Large Language Models The Power of Scale for Parameter-Efficient Prompt Tuning It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners Making Pre-trained Language Models Better Few-shot Learners Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning Towards a Unified View of Parameter-Efficient Transfer Learning
8	Language Model Pretraining	Jan 31th	Slides Video1 Video2	Deep contextualized word representations BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Unified Language Model Pre-training for Natural Language Understanding and Generation ALBERT: A Lite BERT for Self-supervised Learning of Language Representations RoBERTa: A Robustly Optimized BERT Pretraining Approach ERNIE: Enhanced Language Representation with Informative Entities ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Large Language Models
9	Large Language Models	Feb 7th	Slides Video	Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Language Models are Few-Shot Learners LaMDA: Language Models for Dialog Applications Language Models are Unsupervised Multitask Learners Evaluating Large Language Models Trained on Code PaLM: Scaling Language Modeling with Pathways Llama 2: Open Foundation and Fine-Tuned Chat Models Mixtral of Experts
10	Scaling Law	Feb 7th	Slides Video	Scaling Laws for Neural Language Models Scaling Laws for Transfer Emergent Abilities of Large Language Models Training Compute-Optimal Large Language Models Transcending Scaling Laws with 0.1% Extra Compute Inverse scaling can become U-shaped Are Emergent Abilities of Large Language Models a Mirage?
11	Instruction Tuning & RLHF	Feb 14th	Slides Video1 Video2 Video3	Training language models to follow instructions with human feedback Finetuned Language Models Are Zero-Shot Learners Multitask Prompted Training Enables Zero-Shot Task Generalization LIMA: Less Is More for Alignment Direct Preference Optimization: Your Language Model is Secretly a Reward Model Zephyr: Direct Distillation of LM Alignment
12	Efficient LLM Training	Feb 14st	Slides Video1 Video2	Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation RoFormer: Enhanced Transformer with Rotary Position Embedding Fast Transformer Decoding: One Write-Head is All You Need GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
13	Efficient LLM Inference	Feb 28th	Slides Video1 Video2	BERT Loses Patience: Fast and Robust Inference with Early Exit Confident Adaptive Language Modeling Fast Inference from Transformers via Speculative Decoding Efficient Memory Management for Large Language Model Serving with PagedAttention Flash-Decoding for long-context inference Breaking the Sequential Dependency of LLM Inference Using Lookahead Decoding
14	Compress and Sparsify LLM	Feb 28th	Slides Video	Efficient Large Scale Language Modeling with Mixtures of Experts GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding CoLT5: Faster Long-Range Transformers with Conditional Computation LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale 8-bit Optimizers via Block-wise Quantization QLoRA: Efficient Finetuning of Quantized LLMs BitNet: Scaling 1-bit Transformers for Large Language Models
15	LLM Prompting	Mar 6th	Slides Video	Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Self-Consistency Improves Chain of Thought Reasoning in Language Models Tree of Thoughts: Deliberate Problem Solving with Large Language Models Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks Least-to-Most Prompting Enables Complex Reasoning in Large Language Models Measuring and Narrowing the Compositionality Gap in Language Models ReAct: Synergizing Reasoning and Acting in Language Models Self-Refine: Iterative Refinement with Self-Feedback
(Large) Multimodal Models
16	Vision Transformers	Mar 6th	Slides Video1 Video2	An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Training data-efficient image transformers & distillation through attention Emerging Properties in Self-Supervised Vision Transformers SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications MetaFormer Is Actually What You Need for Vision Masked Autoencoders Are Scalable Vision Learners
17	Diffusion Models	Mar 13th	Slides Video	Maximum Likelihood Training of Score-Based Diffusion Models Score-Based Generative Modeling through Stochastic Differential Equations Denoising Diffusion Implicit Models Denoising Diffusion Probabilistic Models DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps Consistency Models
18	Image Generation	Mar 13th	Slides Video	High-Resolution Image Synthesis with Latent Diffusion Models Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding Scaling Autoregressive Models for Content-Rich Text-to-Image Generation Hierarchical Text-Conditional Image Generation with CLIP Latents PIXART-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis Adversarial Diffusion Distillation
19	Multimodal Model Pre-training	Mar 20th	Slides Video	Learning Transferable Visual Models From Natural Language Supervision BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation CoCa: Contrastive Captioners are Image-Text Foundation Models Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks VinVL: Revisiting Visual Representations in Vision-Language Models Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Sigmoid Loss for Language Image Pre-Training
20	Large Multimodal Models	Mar 20th	Slides Video1 Video2	Flamingo: a Visual Language Model for Few-Shot Learning Visual Instruction Tuning InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning PaLI: A Jointly-Scaled Multilingual Language-Image Model PaLI-3 Vision Language Models: Smaller, Faster, Stronger Generative Multimodal Models are In-Context Learners InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Gemini: A Family of Highly Capable Multimodal Models
Augmenting Foundation Models
21	Tool Augmentation	Mar 27th	Slides Video1 Video2	Toolformer: Language Models Can Teach Themselves to Use Tools ART: Automatic multi-step reasoning and tool-use for large language models ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings AgentBench: Evaluating LLMs as Agents CogAgent: A Visual Language Model for GUI Agents WebArena: A Realistic Web Environment for Building Autonomous Agents
22	Retrieval Augmentation	Mar 27th	Slides Video1	REALM: Retrieval-Augmented Language Model Pre-Training Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Improving language models by retrieving from trillions of tokens Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection REPLUG: Retrieval-Augmented Black-Box Language Models
Project Presentation
23	Final Presentation	Apr 3rd		Group 1 - 10: Presenting their final project. Each with 12 minutes.
24	Final Presentation	Apr 3rd		Group 11 - 20: Presenting their final project. Each with 12 minutes.

University of Waterloo Academic Integrity Policy

The University of Waterloo Senate Undergraduate Council has also approved the following message outlining University of Waterloo policy on academic integrity and associated policies.

Academic Integrity

In order to maintain a culture of academic integrity, members of the University of Waterloo community are expected to promote honesty, trust, fairness, respect and responsibility. Check the Office of Academic Integrity's website for more information. All members of the UW community are expected to hold to the highest standard of academic integrity in their studies, teaching, and research. This site explains why academic integrity is important and how students can avoid academic misconduct. It also identifies resources available on campus for students and faculty to help achieve academic integrity in, and our, of the classroom.

Grievance

A student who believes that a decision affecting some aspect of his/her university life has been unfair or unreasonable may have grounds for initiating a grievance. Read Policy 70 - Student Petitions and Grievances, Section 4. When in doubt please be certain to contact the department's administrative assistant who will provide further assistance.

Discipline

A student is expected to know what constitutes academic integrity, to avoid committing academic offenses, and to take responsibility for his/her actions. A student who is unsure whether an action constitutes an offense, or who needs help in learning how to avoid offenses (e.g., plagiarism, cheating) or about “rules” for group work/collaboration should seek guidance from the course professor, academic advisor, or the Undergraduate Associate Dean. For information on categories of offenses and types of penalties, students should refer to Policy 71-Student Discipline. For typical penalties check Guidelines for the Assessment of Penalties.

Avoiding Academic Offenses

Most students are unaware of the line between acceptable and unacceptable academic behaviour, especially when discussing assignments with classmates and using the work of other students. For information on commonly misunderstood academic offenses and how to avoid them, students should refer to the Faculty of Mathematics Cheating and Student Academic Discipline Policy.

Appeals

A decision made or a penalty imposed under Policy 70, Student Petitions and Grievances (other than a petition) or Policy 71, Student Discipline may be appealed if there is a ground. A student who believes he/she has a ground for an appeal should refer to Policy 72 - Student Appeals.

Note for students with disabilities

The AccessAbility Services Office (AAS), located in Needles Hall, Room 1401, collaborates with all academic departments to arrange appropriate accommodations for students with disabilities without compromising the academic integrity of the curriculum. If you require academic accommodations to lessen the impact of your disability, please register with the AAS at the beginning of each academic term.