Section outline

  • March 26th, Thursday (10:30-12:30)

    Neural language models (NLM)

    • Practical issues: parameter freezing, weight tying, softmax temperature

    Contextual word embeddings

    • Static embeddings vs. contextualized embeddings
    • ELMo
    • BERT: encoder-only model
    • BERT: masked language modeling

    References

    • Jurafsky and Martin, chapter 9
    • Voita, NLP Course | For You (web course): Language Modeling
    • Slides from lecture

    Resources