Section outline

  • March 20th, Wednesday (10:30-12:30)

    Neural language models (NLM)

    • Recurrent NLM: inference (continued)
    • Recurrent NLM: training
    • Practical issues: parameter freezing, weight tying, softmax temperature

    Contextualised word embeddings

    • Transformers: short recap
    • Attention
    • Static embeddings vs. contextualized embeddings
    • ELMo

    References

    • Jurafsky and Martin, section 9.2
    • Jurafsky and Martin, chapter 11
    • Voita, NLP Course | For You (web course): Language Modeling
    • Slides from lecture

    Resources