CS 886: Deep Learning and Natural Language Processing
Winter 2020

Ming Li DC 3355, x84659 mli@uwaterloo.ca
Course time and location:Mondays 3:00-5:50pm, DC 2585 (Starting from Jan 20)
Office hours:I will try to do office hours by phone (please call me at 519-500-3026 any time), or by appointment
Reference Materials:Papers listed below.

Deep learning has brought truly revolutionary changes in NLP research. This course intends to review the recent progress of this exciting development.

The course will be run as follows. I will do some lectures at the beginning introducing recent breakthrough results that have fundamentally changed NLP research. These include word2vec, and pretraining models such as GPT and BERT, and single headed attention RNN. Then during the second part of the course, each student will present one or a group of research papers from the paper list I give below (mainly from the first two lists, and please discuss with me about your choices). The paper you choose should represent an important progress in NLP or on shortcomings of current approachs and how we can solve the fundamental problem in NLP: understanding. Additionally, each student will need to do one course project of your own choice and present it to the class at the end of the term. I expect the students already knew the basics of deep learning such as different type of gates, pooling, backpropogation gradient descent methods, fully connected networks, recurrent networks such as LSTM, convolutional networks, and more specialized structures such as residue networks and Grid LSTM, recursive structure, memory networks, sequence-to-sequence structure, generative adversarial nets (GANs). If you do not already know about these, you can read about these materials online or go to my lecture notes at: https://cs.uwaterloo.ca/~mli/cs898-2017.html

GPUs: In order for some of you to do experiments, students can go to https://www.awseducate.com/application to sign up. Amazon will review the application for a couple of days. More information can be found at: https://aws.amazon.com/cn/education/awseducate/ Sharcnet might be another resource for GPU. It is possible to apply for a TPU from google, https://heartbeat.fritz.ai/step-by-step-use-of-google-colab-free-tpu-75f8629492b3

Marking Scheme: Each student is evaluated according to the following three components:

Presentations and relevant papers will be posted on this website (the presenters should provide these materials to me) several days before class.

Course announcements and lecture notes will appear on this page. Please look at this page regularly.

    Reading Materials:

Lecture Notes:


This Monday (Jan. 13)'s class is moved to Jan 17, 5pm, same classroom.

We will move to DC 2585, a larger classroom, starting from the Jan 20's class. The class time will be 3pm to 5:50pm, Mondays. This way, all qualified students in the waiting list will be able to enroll to this course.

For all those who would like to be added to this course, please come to class today (Jan 20) and give me your student ID.

Announcement Jan 26, 2020. Attention: If you have visited a city in China in the past 10 days, please wear a mask to attend the class, to protect other students.

Announcement Feb. 3, 2020: If you had contact with anybody who might have had contact with people with nCoV in the past 15 days or if you have any symptoms, please do not come to the class (and I do not need a doctor's note). This class will no longer have the attendance requirement. Everybody will have that 5% attendance mark automatically. Of course, if you stay home, then please read thru the presented materials on your own. You are expected to know the material.

It's time to talk to me about your course (research) project. Note, this is independent to your presentation (you can of course extend the paper you have presented).

Please make sure you send me your presentation ppt or pdf file 1 day before the presentation.

The final projects are due on April 10th, via emailing me in pdf. (Please submit your 10 minute presentation together with the final project.)

The final project presentations will be on March 30 and April 1. 10 minutes each person. Please let me know which day you like to present (first come first choose). You will use your own laptop to present. As the time will be very tight, please test your computer connection beforehand.

Announcement March 13, 2020. Due to school corona virus closure, We will stop the last day of presentation for March 16. Please see the detailed arrangement below. The 5 class attendance mark will now depend on your two half-page reviews of two presentations of final projects, respectively.

Announcement March 21, 2020. Course evaluation, attention all students. Please go to Evaluation Website to evaluate our course CS886 Section 002 (SEM), open Sun. Mar 22 11:59 to Fri. Apr 3, 11:59pm. I wish you are all safe!

Announcement March 22, 2020. If any student have any question, please call me any time at: 519-500-3026. Stay safe!

Announcement March 26, 2020. Deadlines postponed The new deadline for submitting the final project report is April 15 (any time in that day). On the same day, submit your voice-over-ppt (10 minute) presenting your project. You no longer need to write reviews for two other student presentations.

Announcement March 28, 2020. GPU's Several students asked about GPU resources. There are 3 GPU servers: gpu1, gpu2, and gpu3. First ssh to datasci.cs.uwaterloo.ca, then using your linux.student.cs userid and password. From there, ssh to one of the GPU servers. But I think these are not sufficent to train big models like BERT. Please use these only for light trainings.

Announcement April 11, 2020 April 15 is the deadline for submitting the project report (approximately 10 pages) and voice-over-ppt (10 minutes). All projects and voice-over-ppt will be posted on this website unless you explicitly tell me not to do so with a good reason (for example, the material will be published in a paper).

Announcement April 15, 2020 Today is the deadline. Please email me your project report and voice-over-ppt. I will acknowledge each email. For submission: please send email attaching 2 files, if possible. Do not do anything else such , as zip etc, to make things easier on my side.

Announcement April 16, 2020 As many people do not wish to publicize their work because they wish to continue their work as a part of larger research programs, I decide we will not publish anybody's ppt and final project.

Announcement April 26, 2020 This is our last announcement of the class. I have now finished marking all the reports and watched all presentations. I will input these marks to the university system. I would like to take this chance to thank every student in this class. COVID-19 stopped me from saying good-bye to you, I wish all of you stay healthy and carry on to do great research in NLP and deep learning.

Presentation Schedule:

Final Project Presentation (10 Minutes Each Person):