University Homepage School Homepage School Contacts School Search

Machine Learning Practical (MLP) 2016-17

[course descriptor]

News

• A few examples of good reports submitted for coursework 4
• MLP Coursework 4 deadline extended to Tuesday 21 March
• Ben Allison (Amazon) will give the guest lecture on 15 February, talking about Building Production Machine Learning Systems
• Benigno Uria (DeepMind) will give the guest lecture on 8 February
• Lecture slot for 1 February will be for questions and answers, and discussion. I am happy to discuss in more depth topics from the lectures in semester 1.
• Courseworks 3 and 4 now available. Along with notebooks 09a, Object recognition with CIFAR-10 and CIFAR-100, and 09b, Music genre classification with the Million Song Dataset.
• Slides from Peter Bell's guest lecture on curriculum, multi-task, and student-teacher learning.
• MLP Piazza page at https://piazza.com/ed.ac.uk/spring2017/infr11132/home
Slides and lecture recording introducing semester 2 of MLP.
• Eighth lab - Introduction to TensorFlow - now available
• Office hours: 2-4pm, IF-3.15, every Friday, starting week 2 (27 January).
• Sign-up for labs at http://doodle.com/poll/dy78erdey4twhk7c. You should attend one 1-hour scheduled lab session per week. All labs will take place in the same place as semester 1, Forrest Hill 3.D01, but some of them will be at different times. Labs will start in week 2 (week of 23 January).
• There will be 5 lectures in semester 2 (weeks 1-5), taking place in Appleton Tower, LT1 at the usual time 11:00 on Wednesdays
  The first lecture will be an introduction to semester 2 of MLP; the remaining four lectures will be guest lectures to give you some introduction to state-of-the-art work in the area.
• Mid-point Course survey - closes 23:59 on Monday 16 January
• Seventh lab on autoencoders now available
• Note on Jupyter Notebook over ssh
• Second coursework released. Submit by 16:00 on 24 November
• Sixth lab on dropout and maxout now available
• Mid-semester course feedback - summary and response
• Fifth lab on relu, tanh, L1/L2 regularisation, and data augmentation now available
• Fourth lab on generalisation and overfitting now available
• First coursework released. Submit by 16:00 on 27 October
• Office hours 4-5pm, Fridays, IF-3.15.
• Third lab, on softmax, multi-layer networks, and backprop, now available. (Instructions to get a new branch.)
• Apologies for the low screen capture quality in the recording of lecture 3 (technical error...)
• Second lab now available.
• Online questions, answers, discussion of the lecture slides and lab material at the MLP Forum which uses nb - please sign up.
• Lecture recordings now available
• Lab material available using GitHub at https://github.com/CSTR-Edinburgh/mlpractical.
• If you have a problem with insufficient quota space when setting up the environment on DICE, please see this note.
• Sign-up for labs at http://doodle.com/poll/kf7yhrvy27bvmqbc. You should attend one 1-hour scheduled lab session per week. All labs will take place in Forrest Hill, room 3.D01. If you sign up for a 1-2pm lab, please be prepared to leave by 2pm, so that people can participate in the following lab.
• Lectures from week 2 of semester 1 will take place in Basement Lecture Theatre, Adam House, 3 Chambers Street
• First lecture only (21 September) will take place in room G.07/G.07a in the Informatics Forum (ground floor)
• The Machine Learning Practical is now a 2-semester 20-credit course.

Introduction

The coursework-based Machine Learning Practical (MLP) is focused on the implementation and evaluation of machine learning systems. Students who do this course will have experience in the design, implementation, training, and evaluation of machine learning systems.

In 2016-17 the course will focus on deep neural networks. There are some changes compared with the 2015-16 MLP course, most notably the course is now 20 credits, and runs through both semesters.

MLP requires mathematical ability (calculus, linear algebra, and probability) and programming ability (the course will be based on Python using Numpy). Some previous experience of machine learning is is extremely helpful.
Undergraduates: If you have taken Informatics 2B and IAML, and can program, you are qualified to do the course.
It is also recommended to take MLPR.

MLP 2016-17: Deep Neural Networks

This year the MLP course will focus on deep neural networks. During semester 1 we shall focus on the classification of handwritten digits using the well-known MNIST dataset. Using a Python software framework that we shall provide, and a series of iPython notebooks, the aim of the course is to train multi-layer neural neural network classifiers and convolutional network classifiers to address this handwritten digit classification problem. There will be a series of nine weekly lectures in semester 1 to provide the required theoretical support to the practical work.

In semester 2, we shall focus on using deep neural networks to address a second task. In semester 2 there will be a series of five weekly lectures (weeks 1-5) to support the coursework, and also to provide insights to the current state of the art in this very fast moving area.

Schedule

• Lectures: Wednesdays, 11:00, Adam House, 3 Chambers Street - Basement Lecture Theatre.
  
• Labs: A number of weekly lab sessions will be scheduled, students are expected to attend one of these.
  
• Lecturer:
• Teaching assistant: Matt Graham
• Demonstrators:
  • Semester 1: Simão Eduardo, Xingxing Zhang
  • Semester 2: Yang Liu, Lazar Valkov, Pol Moreno Comellas
• Markers: Joachim Fainberg, Valentin Radu, Sam Ribeiro

Lectures

Semester 1 (9 lectures)

All lectures at 11am on Wednesdays
Lectures in Basement Lecture Theatre, Adam House, 3 Chambers Street.
Lecture recordings.

1. Wednesday 21 September 2016 (Informatics Forum, G.07/G.07A). Introduction to MLP. Linear networks. Gradient descent.
   Slides.
   Lecture recordings: Part 1 - Introduction;   Part 2 - Single Layer Networks.
    
2. Wednesday 28 September 2016. Stochastic gradient descent. Classification.
   Slides.
   Lecture recordings: Part 1 - Stochastic Gradient Descent;   Part 2 - Classification, Sigmoid, and Softmax.
    
3. Wednesday 5 October 2016. Multi-layer networks.
   Slides.
   Lecture recordings: Part 1 - Classification with Neural Networks - Templates and Features;   Part 2 - Learning Multi-layer Networks.
    
4. Wednesday 12 October 2016. Adaptive learning rates. First coursework. Generalisation.
   Slides.
   Lecture recordings: Part 1 - Learning rates and momentum; coursework 1;   Part 2 - Generalisation in practice.
    
5. Wednesday 19 October 2016. Regularisation. Hidden Unit Transfer Functions.
   Slides.
   Lecture recordings: Part 1 - Regularisation;   Part 2 - Hidden unit transfer functions.
    
6. Wednesday 26 October 2016. Normalisation. Initialisation. Pretraining.
   Slides.
   Lecture recordings: Part 1 - Feature normalisation, random initialisation, batch normalisation;   Part 2 - Stacked pretraining.
    
7. Wednesday 2 November 2016. Introduction to coursework 2; Convolutional networks (1).
   Slides.
   Lecture recordings: Part 1 - Introduction to coursework 2;   Part 2 - Convolutional neural networks.
    
8. Wednesday 9 November 2016. Convolutional networks (2).
   Slides.
   Lecture recordings: Part 1 - Gradient descent training of convolutional networks;   Part 2 - Recent advances usng convolutional networks.
    
9. Wednesday 16 November 2016. Recurrent neural networks.
   Slides.
   Lecture recordings: Part 1 - Introduction to recurrent neural networks, backprop through time, LSTM.;   Part 2 - Applications of RNNs.

Semester 2 (5 lectures)

All lectures at 11am on Wednesdays, Appleton Tower LT1.

1. Wednesday 18 January 2017. Introduction to part 2 of MLP.
   Slides.
   Lecture recordings: MLP in Semester 2.
2. Wednesday 25 January 2017. Multi-task learning and related supervised methods. Peter Bell.
   Slides.
   Lecture recordings: Multi-task learning and related supervised methods.
3. Wednesday 1 February 2017. Discussion lecture.
4. Wednesday 8 February 2017. Benigno Uria (DeepMind).
5. Wednesday 15 February 2017. Ben Allison (Amazon), "Building Production Machine Learning Systems"

You can discuss and ask questions about these lectures on the online MLP Forum.

Labs

The lab material is available using GitHub at https://github.com/CSTR-Edinburgh/mlpractical. You do not require a github login to use this. Each new lab corresponds to a new branch in git: see Getting started in a Lab for details about this.

Labs use iPython and Jupyter notebook. There are many Python/numpy tutorials on the web, I think that this is a good one: http://cs231n.github.io/python-numpy-tutorial/.

Solutions for each week's lab notebook will be included in the corresponding notebook in the branch you checkout with the following week's material.

• Lab 8: Introduction to TensorFlow
• Lab 7: Autoencoders
• Lab 6: Dropout and maxout
• Lab 5: tanh, relu, L1/L2 regularisation, and data augmentation
• Lab 4: Generalisation and overfitting
• Lab 3: Softmax, multi-layer networks, and backprop
• Lab 2: Single-layer networks and visualising trajectories in parameter space
• Lab 1: Getting started and data providers
• Instructions to get a new branch
• Setting up your environment

Lab Sessions

You should attend one 1-hour scheduled lab session per week.
Semester 2: All labs will take place in Forrest Hill, room 3.D01. Labs will take place on Mondays (9-10am; 10-11am), Tuesdays (1-2pm; 2-3pm), and Thursdays (9-10am; 10-11am).
Semester 1: All labs will take place in Forrest Hill, room 3.D01. Labs will take place on Mondays (1-2pm; 2-3pm), Tuesdays (1-2pm; 2-3pm), and Thursdays (1-2pm; 2-3pm).

Office hours

MLP Office Hour from 4-5pm on Fridays, 3.15 in the Informatics Forum, (7 October - 18 November) If you have a question about the course, then you are welcome to come along and ask/discuss. If your question is about the software, the environment, or implementation issues relating to the labs or coursework, then please ask your questions in the lab, rather than the office hour.

Coursework

• Coursework 1: Available 10 October 2016, Submit 27 October 2016, Feedback 10 November 2016. Worth 10%.
  

  This coursework is concerned with training multi-layer networks to address the MNIST digit classification problem. It builds on the material covered in the first three lab notebooks and the first four lectures. It is highly recommended that you complete the first three lab notebooks before starting the coursework. The aim of the coursework is to investigate the effect of learning rate schedules and adaptive learning rates on the progression of training and the final performance achieved by the trained models.

  The code for the coursework is available on the course Github repository on a branch mlp2016-7/coursework1. Full details in the Coursework 1 document.

  13-October-2016: There is a small error in the code that was distributed for the first coursework. In the file mlp/learning_rules.py in the reset method of the MomentumLearningRule the line (number 145 in the file)

  for mom in zip(self.moms):

  should instead be

  for mom in self.moms:

  This causes the reset method to not set all of the momentum arrays to zeros as intended. The code on the Github repository has now been updated.

• Coursework 2: Available 2 November 2016, Submit 24 November 2016, Feedback 8 December 2016. Worth 25%
  

  The aim of this coursework is to use a selection of the techniques covered in the course so far to train accurate multi-layer networks for MNIST classification. It is intended to assess your ability to design, implement and run a set of experiments to answer specific research questions about the models and methods covered in the course.

  You should choose three different topics to research. Our recommendation is to choose one simpler question and two which require more in-depth implementations and/or experiments.

  Note that it is in your interest to start running the experiments for this coursework as early as possible. Some of the experiments may take significant compute time.

• Courseworks 3 and 4: Available 27 January 2017.
  Courseworks 3 and 4 take the form of a mini-project with the aim to explore deep learning and neural network techniques for classification using \emph{one} of two datasets: CIFAR-10 and CIFAR-100 (object recognition in images); or the Million Song Dataset (music genre recognition). The coursework is supported by notebooks 09a, Object recognition with CIFAR-10 and CIFAR-100, and 09b, Music genre classification with the Million Song Dataset. You should treat courseworks 3 and 4 as single mini-project, using the same dataset and task throughout, for which you submit two reports -- an initial report (coursework 3) and a final report (coursework 4).
  • Coursework 3: Submit 16 February 2017, Feedback 6 March 2017. Worth 25%
  • Coursework 4: Submit 21 March 2017 (deadline extended), Feedback 6 April 2017. Worth 40%
  Coursework 4 FAQ.
  Examples of good reports submitted for Coursework 4.

Please make sure you have read and understood

• the late coursework policy, and
• the academic conduct policy

Reading

Textbooks

• Michael Nielsen, Neural Networks and Deep Learning, 2015. This free online book has excellent coverage of feed-forward networks, training by back-propagation, error funcions, regularisation, and convolutional neural networks. It uses the MNIST data as a running example. This book also comes with a python codebase. Do not use this codebase in your practical work, use the one we provide! - The aim of this course is to enable you to do your own implementation. However, this book is an excellent resource and is highly recommended.
• Yoshua Bengio, Ian Goodfellow and Aaron Courville, Deep Learning, 2015, Book in preparation for MIT Press. Chapters 6-9 are most relevant.
• Christopher M Bishop, Neural Networks for Pattern Recognition, 1995, Clarendon Press.

Additional material

• Learning Rate Schedules: A Senior et al An empirical study of learning rates in deep neural networks, Proc IEEE ICASSP, 2013.
• Maxout: I Goodfellow et al Maxout networks, Proc ICML 2013.
• Denoising autoencoders: P Vincent et al, Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion, JMLR, 11:3371--3408, 2010.
• Y LeCun etl Efficient backprop, Neural Networks: Tricks of the Trade, Springer LNCS 1524, 1998.
• Convolutional Networks: Y LeCun et al Gradient-Based Learning Applied to Document Recognition, Proc IEEE, 1998.

Review articles (including material not covered in this course)

• Y Bengio, Learning deep architectures for AI, Foundations and trends in Machine Learning, 2009
• G Hinton, Boltzmann machine, Scholarpedia,2007.
• Y Lecun et al, A Tutorial on Energy-Based Learning, To appear in "Predicting Structured Data", MIT Press, 2006.

Course Feedback

We are very keen to hear any feedback on MLP - what are we doing right? what could be done better? what might we consider introducing? Contact me any time if you have any comments.

There is also an official feedback form near the end of each semester, please do take the time to complete it. Mid-semester, there is also a briefer feedback opportunity.

• October 2016: mid-semester feedback - summary and response.
• November/December 2015: course survey for last year's course (2015/16) - report; response.

---

This page maintained by Steve Renals.
Last updated: 2017/09/15 16:37:07UTC