Natural Language Processing
A.Y. 2024/2025
Learning objectives
The course provides an extensive and in-depth introduction to the state of the art and the main research trends in Natural Language Processing (NLP). In particular, the course focus on deep learning methods for NLP, with a specific attention on large language models. Students will deal with fundamental tasks such as syntactic, semantic, and discourse analysis, as well as methods to solve these tasks. A specific focus will be on transfer learning methods and model architectures to address concrete tasks such as text classification, question answering, automatic translation and text generation. These goals will be pursued by a combination of theory, seminars on recent papers and methods, and practical examples. The program is intended for graduate students in computer science and data science who are familiar with machine learning basics. An intruduction to deep learning and neural networks will be provided together with a practical introduction to PyTorch. Coding in Python will play also an important role in the classes.
Expected learning outcomes
Through reading recent research papers, programming assignments, and a final project, students will acquire the following skills: 1) knowing and understanding the main topics as well as the research issues and the future trends in the field of Natural Language Processing (NLP); 2) learn how to apply NLP methods to a corpus of texts for a specific need; 3) being able to judge the quality of different design and implementation choices when coming to a NLP project; 4) being able to design, implement, and evaluate a specific project focused on NLP tasks; 5) understand the notion of language model and being able to detect language specificities and topics in a corpus of text documents; 6) being able to use the Python stack of libraries and tools required to develop a NLP project.
Lesson period: Second semester
Assessment methods: Esame
Assessment result: voto verbalizzato in trentesimi
Single course
This course can be attended as a single course.
Course syllabus and organization
Single session
Responsible
Lesson period
Second semester
Course syllabus
The lectures provide an in-depth introduction to the main research topics in the field of Deep Learning applied to Natural Language Processing. In addition to the lectures, there is a final project through which students will acquire the necessary skills to design, implement and understand the main neural network models for natural language, using Python and Pytorch.
INTRODUCTION TO NLP
Introduction to the lectures topics, exam modality and scheduling. Preliminary tokenization and frequency-based indexing.
VECTOR SPACE MODEL, EVALUATION AND RELEVANCE FEEDBACK
Scoring, term weighting and the vector space model. Advanced text pre-processing and weighting. Evaluation in information retrieval. Relevance feedback and query expansion. Improving performances by relevance feedback.
UNSUPERVISED AND SUPERVISED TEXT CLASSIFICATION AND TOPIC MODELING
Matrix decompositions and Topic Models. Implementing and using LDA. Zero-shoot learning of topics. Supervised text classification. Use case on text classification. Introduction to deep learning classification models.
INTRODUCTION TO DEEP LEARNING FOR NLP
Backpropagation and Feedforward Neural Networks. Word and document vectors. Word2Vec.
LANGUAGE MODELS
Recurrent Neural Networks and Language Models. Vanishing Gradients, Seq2Seq learning. LSTM and GRU models. Encoder-Decoder architectures for sequence learning. Autoencoders.
ATTENTION AND TRANSFORMERS
Attention mechanism in the encoder-decoder architecture. Some transformer models: BERT. GPL2.
APPLICATIONS
Natural Language Generation. Integrating knowledge bases in language models.
INTRODUCTION TO NLP
Introduction to the lectures topics, exam modality and scheduling. Preliminary tokenization and frequency-based indexing.
VECTOR SPACE MODEL, EVALUATION AND RELEVANCE FEEDBACK
Scoring, term weighting and the vector space model. Advanced text pre-processing and weighting. Evaluation in information retrieval. Relevance feedback and query expansion. Improving performances by relevance feedback.
UNSUPERVISED AND SUPERVISED TEXT CLASSIFICATION AND TOPIC MODELING
Matrix decompositions and Topic Models. Implementing and using LDA. Zero-shoot learning of topics. Supervised text classification. Use case on text classification. Introduction to deep learning classification models.
INTRODUCTION TO DEEP LEARNING FOR NLP
Backpropagation and Feedforward Neural Networks. Word and document vectors. Word2Vec.
LANGUAGE MODELS
Recurrent Neural Networks and Language Models. Vanishing Gradients, Seq2Seq learning. LSTM and GRU models. Encoder-Decoder architectures for sequence learning. Autoencoders.
ATTENTION AND TRANSFORMERS
Attention mechanism in the encoder-decoder architecture. Some transformer models: BERT. GPL2.
APPLICATIONS
Natural Language Generation. Integrating knowledge bases in language models.
Prerequisites for admission
Intermediate knowledge of Python. Basic knowledge of derivatives and understanding of matrix/vector notation and operations. Basics of probabilities and gaussian distributions.
Teaching methods
The course is given in the form of lectures with extensive use of examples and support materials such as Python notebooks. Slides and handouts are employed throughout the lectures and they are progressively published on the reference course website on the Ariel platform and on the GitHub repository (https://github.com/afflint/nlp).
Lecture attendance is not mandatory, but it is strongly recommended.
Lecture attendance is not mandatory, but it is strongly recommended.
Teaching Resources
- Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval (Vol. 1, p. 496). Cambridge: Cambridge University Press. (Http://nlp.stanford.edu/IR-book/)
- Notes, notebooks and materials provided by the lecturer and published on the website of the course (https://aferrarair.ariel.ctu.unimi.it)
- Notes, notebooks and materials provided by the lecturer and published on the website of the course (https://aferrarair.ariel.ctu.unimi.it)
Assessment methods and Criteria
Development of a project. The project topic has to be previously discussed with the lecturer. The project should demonstrate the comprehension of the lectures topics and the capability of proposing and motivating innovative solutions to specific research problems.
The project will be evaluated through a discussion with the lecturer about the project outcomes and the related topics. The evaluation will take into account both the project and the interview.
Using the SIFA service for participating in the examination is mandatory. After the registration to an examination on SIFA, the students are requested to contact the lecturer for scheduling the discussion.
The project will be evaluated through a discussion with the lecturer about the project outcomes and the related topics. The evaluation will take into account both the project and the interview.
Using the SIFA service for participating in the examination is mandatory. After the registration to an examination on SIFA, the students are requested to contact the lecturer for scheduling the discussion.
INF/01 - INFORMATICS - University credits: 6
Lessons: 48 hours
Professor:
Ferrara Alfio
Shifts:
Turno
Professor:
Ferrara AlfioProfessor(s)
Reception:
On appointment. The meeting will be online by first contacting the professor by email.
Online. In case of a meeting in person, Department of Computer Science, via Celoria 18 Milano, Room 7012 (7 floor)