University Homepage School Homepage School Contacts School Search

Lecture 12 - Language Modelling

In this lecture we discussed n-gram and neural network language models.

n-gram language models (LMs)

• The basic model, and underlying assumptions (approximatinmg conditional probabilities using finite context)
• Maximum likelihood estimation of n-gram probabilities from text data
• Smoothing and the zero probability problem:
  • Discounting approaches
  • Back-off models
  • Interpolating multiple n-gram models
• Perplexity measure

Neural network LMs

• Distributed representations
• Feed-forward Neural Predictive LM - NPLM (Bengio)
  • Architecture
  • Training
  • Comparison with n-grams
• Recurrent Neural Network LM - RNNLM (Mikolov)
  • Architecture
  • Comparison with NPLM
• Reducing computation at the output layer
  • Vocabulary size, large output layers, and computational problems
  • Shortlist - do not include less frequent words at the output
  • Class-based factorisation and hierarchical softmax
  • Noise contrastive estimation (NCE)
• State-of-the-art NN LMs - Google 2016 systems

---

This page maintained by Steve Renals.
Last updated: 2017/03/06 14:17:52UTC