NATURAL LANGUAGE PROCESSING 2024-2025 - INQ0091105
Topic outline
-
Content: Detailed description of course content and prerequisites can be found here.
Textbook: The adopted textbook is Speech and Language Processing (3rd Edition, draft, January 12, 2025) by Dan Jurafsky and James H. Martin, available here.
Additional resources: The following textbook can be used for consultation only: Introduction to Natural Language Processing by Jacob Eisenstein, October 2019, MIT Press, preprint version available here. The course also uses an electronic forum for discussion of technical matter and administrative information. You can also access video recordings of the lectures from academic year 2021/22 at this link
Logistics: Lectures are on Monday 16:30-18:30 (room Ce) and on Wednesday 16:30-18:30 (room Ce).
Office hours: Wednesday 12:30-14:30, email appointment required. Meetings can be face-to-face or else on-line at this Zoom link.
-
Forum for general news and announcements. Only the lecturer can post in this forum. Subscription to this forum is automatic for every student who has registered to this course.
-
Forum for discussion of technical matter presented during the lectures. Any student with a unipd account can post in this forum. Subscription to this forum is automatic for every student who has registered to this course.
-
Forum for discussion of technical matter presented during the open laboratory sessions. Any student with a unipd account can post in this forum. Subscription to this forum is automatic for every student who has registered to this course.
-
Forum for project discussion. Any student with a unipd account can post in this forum. Subscription to this forum is automatic for every student who has registered to this course.
-
-
February 24th, Monday (16:30-18:30)
Course administration and presentation
- Content outline
- Laboratory sessions
- Course requirements
- Textbook
- Project
- Coursework
- Statistics
- Lecturer evaluation
Natural language processing: An unexpected journey
- What is natural language processing?
- A few case studies: finance, social networks, health
- Very short history of natural language processing
- Why is natural language processing tricky?
- Ambiguity, composition, recursion and hidden structure
- How does natural language processing work?
- Learning & knowledge
- Search & learning
- Market, environment and ethics
References
- Slides from the lecture
- Eisenstein, chapter 1 for learning & knowledge and for search & learning
Resources
-
February 26th, Wednesday (16:30-18:30)
Essentials of linguistics
- What is linguistics?
- Phonology
- Morphology
- Part of speech
- Syntax: phrase structure and dependency structure
- Lexical semantics and general semantics
- Pragmatics and discourse
Text normalization
- Regular expressions
- Word types and word tokens
- Corpora
- Language identification and spell checking
- Text normalization: contraction, punctuation and special characters
References
- Slides from the lecture
- Jurafsky and Martin, chapter 2
Resources
-
March 3rd, Monday (16:30-18:30)
Text normalization
- Word tokenization, character tokenization, and subword tokenization
- Byte-pair encoding algorithm
- Sentence segmentation and case folding
- Stop words, stemming and lemmatization
- Research papers
Words and meaning
- Lexical semantics
- Distributional semantics
- Review: vectors
- Term-context matrix
References
- Jurafsky and Martin, chapter 2
- Jurafsky and Martin, chapter 6
- Eisenstein, section 14.3
Resources
-
March 5th, Wednesday (16:30-18:30)
Words and meaning
- Pointwise mutual information
- Probability estimation
- Examples
- Practical issues
- Truncated singular value decomposition
- Neural word embeddings
- Word2vec and skip-gram
- Logistic regression
References
- Jurafsky and Martin, chapter 6
- Voita, NLP Course | For You (web course): Word embeddings
-
March 10th, Monday (16:30-18:30)
Words and meaning
- Training
- Practical issues
- FastText and GloVe
- Semantic properties of neural word embeddings
- Evaluation
- Cross-lingual word embeddings
- Research papers
Language models
- Language modeling: word prediction and sentence distribution
- Language modeling applications
- Relative frequency estimation
- N-gram model
References
- Jurafsky and Martin, chapter 6
- Jurafsky and Martin, chapter 3
Resources
-
March 12th, Wednesday (16:30-18:30)
Language models
- N-gram probabilities and bias-variance trade-off
- Practical issues
- Evaluation: perplexity measure
- Sampling sentences
- Smoothing: Laplace and add-k smoothing
- Stupid backoff and linear interpolation
- Out-of-vocabulary words
- Limitations of N-gram model
- Research papers
Exercises
- Subword tokenization: BPE algorithm
References
- Jurafsky and Martin, chapter 3
-
March 17th, Monday (16:30-18:30)
Neural language models (NLM)
- General architecture for NLM
- Feedforward NLM: inference
- Feedforward NLM: training
- Recurrent NLM: inference
- Recurrent NLM: inference (continued)
- Recurrent NLM: training
- Practical issues: parameter freezing, weight tying, softmax temperature
References
- Voita, NLP Course | For You (web course): Language Modeling
- Jurafsky and Martin, sections 7.6, 7.7
- Jurafsky and Martin, section 8.2
Resources
-
March 19th, Wednesday (16:30-18:30)
Transformers: short recap
- Attention
- Encoder
- Decoder
- Residual stream
Contextualised word embeddings
- Static embeddings vs. contextualized embeddings
- ELMo
- BERT: encoder-only model
- Masked language modeling
- Next sentence prediction
- GPT: decoder-only model
- Sentence BERT
References
- Jurafsky and Martin, chapter 9
- Jurafsky and Martin, sections 11.1, 11.2, 11.3
- Voita, NLP Course | For You (web course): Language Modeling
Resources