Architectures for big data
A.A. 2022/2023
Obiettivi formativi
The course aims at describing the big data processing framework, both in terms of methodologies and technologies. Part of the lessons will focus on Apache Spark and distributed patterns.
Risultati apprendimento attesi
Students will learn:
How to distribute computation over clusters using Map Reduce model
How to write Apache Spark code
How Hadoop works and why it works that way
What a software architecture is
How to design batch architectures to manage data workflows
Several design patterns that could be used in a distributed environment
The limit of traditional SQL with Big Data
How to distribute computation over clusters using Map Reduce model
How to write Apache Spark code
How Hadoop works and why it works that way
What a software architecture is
How to design batch architectures to manage data workflows
Several design patterns that could be used in a distributed environment
The limit of traditional SQL with Big Data
Periodo: Primo semestre
Modalità di valutazione: Esame
Giudizio di valutazione: voto verbalizzato in trentesimi
Corso singolo
Questo insegnamento non può essere seguito come corso singolo. Puoi trovare gli insegnamenti disponibili consultando il catalogo corsi singoli.
Programma e organizzazione didattica
Edizione unica
Periodo
Primo semestre
Programma
Argomenti principali:
a. Enterprise Architectures
b. Design Patterns
c. Hadoop
d. Distributed Algorithms
e. Big Data and SQL
f. Document Based DB
g. Containers
Tecnologie
a. Python
b. Apache Spark - Resilient Distributed Dataset
c. ELK Stack: Elastic Search, Logstash, Kibana
d. Docker
External Workshops
a. Workshop 1: IOT for Connected Vehicle (Marelli - Riccardo Tomasi, PHD) - TBC
b. Workshop 2: Services and Microservices (artea.com - TBD) - TBC
c. Workshop 3: TBD (Unimi - prof. Dario Malchiodi) - TBC
d. Workshop 4: TBD (Google - TBD) - TBC
a. Enterprise Architectures
b. Design Patterns
c. Hadoop
d. Distributed Algorithms
e. Big Data and SQL
f. Document Based DB
g. Containers
Tecnologie
a. Python
b. Apache Spark - Resilient Distributed Dataset
c. ELK Stack: Elastic Search, Logstash, Kibana
d. Docker
External Workshops
a. Workshop 1: IOT for Connected Vehicle (Marelli - Riccardo Tomasi, PHD) - TBC
b. Workshop 2: Services and Microservices (artea.com - TBD) - TBC
c. Workshop 3: TBD (Unimi - prof. Dario Malchiodi) - TBC
d. Workshop 4: TBD (Google - TBD) - TBC
Prerequisiti
The course requires knowledge of the main topics of bachelor-level computer programming, calculus, probability, and statistics. A knowledge of python could be useful (even if a 101 lecture on how to program with python is part of the lectures calendar)
Metodi didattici
.
Materiale di riferimento
Lectures are based:
· on the notes and sample code published in the calendar of lectures.
It is also suggested to read the following material.
· To practice with Spark: H. Karau, A. Konwinski, P. Wendell, M. Zaharia, Learning Spark. Lightning-Fast Big Data Analysis, O'Reilly, 2015 (ISBN:978-1-449-35862-4).
· For a deeper study of Spark: S. Ryza, U. Laserson, S. Owen, J. Wills, Advanced Analytics with Spark. Patterns for Learning from Data at Scale, O'Reilly, 2015 (ISBN:978-1-491-91276-8).
· About distributed file systems and the MapReduce paradigm: Yahoo! Hadoop Tutorial (besides Chapter 2 in RU).
· For a deeper study of the practical parts: Data Science and Engineering with Spark program of edX.
· on the notes and sample code published in the calendar of lectures.
It is also suggested to read the following material.
· To practice with Spark: H. Karau, A. Konwinski, P. Wendell, M. Zaharia, Learning Spark. Lightning-Fast Big Data Analysis, O'Reilly, 2015 (ISBN:978-1-449-35862-4).
· For a deeper study of Spark: S. Ryza, U. Laserson, S. Owen, J. Wills, Advanced Analytics with Spark. Patterns for Learning from Data at Scale, O'Reilly, 2015 (ISBN:978-1-491-91276-8).
· About distributed file systems and the MapReduce paradigm: Yahoo! Hadoop Tutorial (besides Chapter 2 in RU).
· For a deeper study of the practical parts: Data Science and Engineering with Spark program of edX.
Modalità di verifica dell’apprendimento e criteri di valutazione
The exam consists of an oral test related to the topics covered in the course. The oral test, is based on the discussion of some topics covered in the course. The evaluation of the oral test, expressed on a scale between 0 and 30, takes into account the level of mastery of the topics, clarity, and language skills.
INF/01 - INFORMATICA - CFU: 6
Lezioni: 48 ore
Docente:
Condorelli Andrea
Siti didattici