Neural Information Processing: Course homepage 2014-2015

This is a course for MSc level students. It runs in semester 2.

This course is majorly overhauled this year.
There has a recent surge of computational architectures inspired by how the brain works, and unlike previously, they actually perform well on certain tasks. After describing rigorous ways to describe neural activity mathematically, we present a number of the architectures recently used to do tasks like object recognition, memories, bayesian reasoning, and planning.

The background needed to successfully take this course is a good grounding in mathematics, particularly with regard to probability and statistics, vectors and matrices. The mathematical level required is similar to that which would be obtained by students who did not have significant difficulties with the courses Mathematics for Informatics 1-4 taken in the first two years of the Informatics undergraduate syllabus. The Neural Computation (NC) course is a helpful but not necessary prerequisite. Biological realism is not as important an objective as in that course. Machine learning courses (LfD, PMR, IAML) will be also useful preparations.

Instructors: Mark van Rossum.
Course Tutor: TBA


TBC: Monday and Thursday 9.00-9.50. G200 Seminar Room 3 (Doorway 3), Medical School Teviot

Course Outline


There will be two assessed assignments worth in total 25%. There will be an exam worth 75%.

First assignment . Deadline XXX.

Second assignment, deadline XXX .

Old assigments for practice: A1:12-13, A1:11-12, A1:10-11, A1:09-10, A1:08-09


Theoretical Neuroscience by P Dayan and L F Abbott (MIT Press 2001) is recommended reading, see also the list of errata.
Natural Image Statistics by Aapo Hyvärinen, Jarmo Hurri, and Patrik O. Hoyer. Full version online. This will be supplemented by papers from the literature.

Week-by-week listing (to be updated)

Week 1

Lectures: Introduction: lecture slides (2x2) , lecture slides (single page)
Neural encoding: lecture slides (2x2) , lecture slides (single page)

Week 2

Lectures: Neural encoding (cont.)

Week 3

Neural decoding: lecture slides, lecture slides (single page)

Week 4

Lectures: Information theory: lecture slides, lecture slides (single page)

Week 5

Information theory (cont.).

Week 6

Innovative teaching week. No lectures.

Week 7

Lectures: Predicting Retinal Ganglion Cell Receptive Fields, slides (single page), slides (4up)
Higher Order Statistics slides (single page), slides (4up),
Emergence of Simple-Cell Receptive Field Properties by Learning a Sparse Code for Natural Images, Olshausen BA, Field DJ Nature, 381: 607-609 (1996)
Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1? Olshausen BA, Field DJ, Vision Research, 37: 3311-3325 (1997).
The "independent components" of natural scenes are edge filters. Anthony J. Bell and Terrence J. Sejnowski, Vision Research 37(23): 3327-3338 (1997). preprint, official paper also available electronically via UoE library (Science Direct)
Web resource: Contrast sensitivity section from Visual Acuity chapter from Psychophysics of Vision (Michael Kalloniatis and Charles Luu) at
Background reading on Fourier analysis: Fourier series, Fourier transform

Week 8

Lectures: Higher Order Statistics cont, Beyond ICA slides (single page), slides (4up).
Further References (details not required for NIP course)
A hierarchical Bayesian model for learning non-linear statistical regularities in non-stationary natural signals, Y. Karklin and M. S. Lewicki, Neural Computation, 17 (2): 397-423, (2005)
Learning higher-order structures in natural images, Y. Karklin and M. S. Lewicki, Network: Computation in Neural Systems, 14: 483-499, (2003)
Topographic Independent Component Analysis, A. Hyvarinen, P.O. Hoyer and M. Inki, Neural Computation, 13(7):1527-1558, (2001)
Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces A. Hyvarinen and P.O. Hoyer, Neural Computation, 12(7):1705-1720, (2000)

Week 9

Lectures: Beyond ICA cont, Evaluating Models of Natural Image Patches slides (single page), slides (4up),
Undirected Graphical Models lecture slides (single page), lecture slides (4up).
Natural Images, Gaussian Mixtures and Dead Leaves, Daniel Zoran and Yair Weiss, NIPS 2012.
Sparse deep belief net model for visual area V2. Honglak Lee, Ekanadham Chaitanya, and Andrew Y. Ng. Advances in Neural Information Processing Systems 20 (2008).
Topographic Product Models Applied To Natural Scene Statistics Osindero, S., Welling, M. and Hinton, G. E. Neural Computation, 18(2) (2006).
Further References (details not required for NIP course)
Natural Image Coding in V1: How Much Use is Orientation Selectivity? Eichhorn J, Sinz FH and Bethge M, PLoS Computational Biology 5(4:e1000336) 1-16 (2009)

Week 10

Lectures: Undirected Graphical Models ctd, Lateral Interactions and Feedback lecture slides (single page), lecture slides (4up)
Predictive Coding in the Visual Cortex. Rao, R. P. N. and Ballard, D. H., Nature Neuroscience, 2(1), 79-87, (1999)
Hierarchical Bayesian inference in the visual cortex. Lee, T.S., Mumford, D., Journal of Optical Society of America, A. . 20(7): 1434-1448 (2003)

Week 11

Lectures: Lateral Interactions and Feedback ctd, Object recognition lecture slides (single page), lecture slides (4up), if there is time, discussion of Olshausen papers below
Robust Object Recognition with Cortex-Like Mechanisms Serre, T., L. Wolf, S. Bilschi, M. Riesenhuber and T. Poggio. IEEE PAMI 29(3) 411-426 (2007)
Further References (not examinable)
How Close Are We to Understanding V1? Olshausen BA, Field DJ, Neural Computation, 17, 1665-1699 (2005).
20 years of learning about vision: Questions answered, questions unanswered, and questions not yet asked. Olshausen BA. In: 20 Years of Computational Neuroscience, Springer (2010)

Week 12

Lectures: Mon 31: Guest lecture by Dr Matthias Hennig on Log-linear models for spike train analysis lecture slides

This page is maintained by Chris Williams and Mark van Rossum .

Home : Teaching : Courses 

Informatics Forum, 10 Crichton Street, Edinburgh, EH8 9AB, Scotland, UK
Tel: +44 131 651 5661, Fax: +44 131 651 1426, E-mail:
Please contact our webadmin with any comments or corrections. Logging and Cookies
Unless explicitly stated otherwise, all material is copyright © The University of Edinburgh