CS 479 | SCS | UW

Revised August 25, 2022

CS 479: Neural Networks

General description

This course surveys computation by models of networks of neurons. It covers a variety of methods for designing and training both feedforward and recurrent neural networks. We will study and implement supervised and unsupervised learning methods, as well as some learning algorithms that conform to the biological constraints of neuroscience. We will examine issues in neural learning, such as generalizability and adversarial inputs, and survey some mitigating measures. Throughout the course, discussions will address the relationship of artificial neural networks and neuroscience.

Logistics

Audience

  • Students interested in artificial intelligence, or neuroscience

Normally available

  • Winter or Fall

Related courses

  • Pre-requisites: (One of CS 335, CS 370, CS 371/AMATH 242), (One of STAT 230, STAT 206 with at least 60%, STAT 240)
  • Successors: None
  • Co-requisites: None
  • Anti-requisites: None
For official details, see the UW calendar.

Software/hardware used

  • Python 3
  • Pytorch
  • Jupyter notebook

Typical reference(s)

  • P Dayan and L.F. Abbott, Theoretical Neuroscience, 2001, Massachusetts Institute of Technology Press
  • M Neilsen, Neural Networks and Deep Learning , 2017, Determination press
  • I Goodfellow, Y Bengio, and A Courville, Deep Learning , 2016, Massachusetts Institute of Technology Press
  • C Eliasmith and C Anderson, Neural Engineering , 2003, Massachusetts Institute of Technology Press

Required preparation

At the start of the course, students should be able to
  • Program in a high-procedural programming language and use control instructions, such as loops and conditional statements, basic datatypes, and simple data structures, such as arrays.
  • Calculate derivatives, partial derivatives, Taylor series, max/min of multivariate functions, and other basic concepts encountered in calculus.
  • Use matrix operations such as inversion and multiplication.
  • Compute conditional probabilities using Bayes' theorem.

Learning objectives

At the end of the course, students should be able to
  • Write a program to simulate the activity of a network of neurons
  • Formulate a neural learning method as gradient-based optimization
  • Derive a neural learning method based on energy minimization
  • Encode and decode data from the activity of a population of neurons
  • Evaluate implementation, designs, and optimization strategies
  • Identify some commonalities between artificial neural networks and the brain

Typical syllabus

Networks of neurons (4 hours)
  • Neuron models, spiking vs firing-rate
  • Activation curves
  • Synapses
  • Networks of neurons
  • Overview of Learning: Supervised/Unsupervised/Reinforcement
Supervised learning (9 hours)
  • Training, Validation, Testing
  • Universal Approximation Theorem
  • Cost functions
  • Gradient descent
  • Error Backpropagation
  • Automatic differentiation
  • Overfitting and generalizability (regularization)
  • Optimization considerations (vanishing gradients, SGD)
  • Convolutional neural networks
  • Batch normalization
Unsupervised learning (6 hours)
  • Autoencoders
  • Variational autoencoders
  • Hopfield Networks, Hopfield Energy
Population coding (7 hours)
  • Optimal linear Decoding
  • Transformations, dyynamics
  • Online learning
Recurrent neural networks (3 hours)
  • Back prop through time (BPTT)
  • Long short-term Memories (LSTM)
  • Legender delay networks (time permitting)
Adversarial attacks (2 hours)
  • Targeted vs untargeted
  • FGSM
  • TRADES
Additional topics (5 hours)
  • Vector Embeddings (word2vec)
  • GANs
  • Biological backprop models