The course aims at providing: • Knowledge of the main problems and opportunities arising in the big data context and technological characteristics of the infrastructures and distributed systems used to deal with big data (e.g., Hadoop and Spark). • Ability to write distributed programs to process and analyze big data using programming paradigms based on Map Reduce and Spark (Spark RDD- and DataFrame-based programming). • Knowledge of the (relational and non-relational) database systems that are used to store big data • Ability to implement scalable data analytics processes based on data mining and machine learning algorithms for big data applications.

Python language, basic knowledge of Java language, and basic knowledge of traditional database concepts (relational model and SQL language).

Lectures (45 hours) • Introduction to Big data: characteristics, problems, opportunities (3 hours) • Hadoop and its ecosystem: infrastructure and essential components (3 hours) • Map Reduce programming paradigm (9 hours) • Spark: Spark Architecture, RDD-based and Spark SQL-based programming (16 hours) • Streaming data analysis: Spark Streaming (6 hours) • Data mining and Machine learning libraries: Spark MLlib (6 hours) Laboratory activities (15 hours) • Developing of applications by means of Hadoop and Spark (15 hours)

The course consists of Lectures (45 hours) and Laboratory sessions (15 hours). The laboratory sessions are focused on the course's main topics (Map Reduce, Spark, and MLlib). The Laboratory sessions allow experimental activities on the most widespread open-source products.

Copies of the slides used during the lectures, examples of written exams and exercises, and manuals for the activities in the laboratory will be made available. All teaching material is downloadable from the course website or the Portal. Reference books: • Matei Zaharia, Bill Chambers. Spark: The Definitive Guide (Big Data Processing Made Simple). O'Reilly Media, 2018. • Advanced Analytics and Real-Time Data Processing in Apache Spark. Packt Publishing, 2018. • Tom White. Hadoop, The Definitive Guide. (Third edition). O'Reilly Media, 2015. • Matei Zaharia, Holden Karau, Andy Konwinski, Patrick Wendell. Learning Spark (Lightning-Fast Big Data Analytics). O’Reilly, 2015.

Lecture slides; Text book; Exercises; Exercise with solutions ; Lab exercises; Lab exercises with solutions; Video lectures (current year);

Exam: Computer-based written test in class using POLITO platform;

The exam aims at assessing (i) the ability of the students to write distributed programs to process and analyze big data by means of novel programming paradigms and frameworks (the Map Reduce programming paradigm and the Spark RDD-based programming paradigm) and (ii) the knowledge of the students of the main issues related to the big data topic and the technological infrastructures and distributed systems, including scalable relational and non-relational databases systems, that are used to deal with big data. The exam consists of a written onsite test (with Exam+Lockdown platforms on the student notebook) that lasts 1.5 hours. Specifically, the written onsite test is composed of two parts: - 1-3 programming exercises (structured as open questions) based on MapReduce- and Spark-based programming to be solved using the Java language (max 27 points) - 1-3 multiple choice questions on all the topics addressed during the course (max 6 points). The programming exercises aim to evaluate the ability of the students to write distributed programs to analyze big data through the programming paradigms introduced in the course. The multiple-choice questions are used to evaluate the knowledge of the theoretical concepts of the course and, in particular, the knowledge of the characteristics of the main technological infrastructures and distributed systems (Hadoop and Spark), including scalable relational and non-relational database systems that are used to deal with big data. The evaluation of the programming exercises is based on the correctness and efficiency of the proposed solutions. The exam is closed book. - Books, notes, and any other paper material are not allowed. - Electronic devices of any kind (PC, laptop mobile phone, calculators, etc.), apart from the PC used to take the test, are not allowed. The exam is passed if the mark of the written exam is greater than or equal to 18 points.

In addition to the message sent by the online system, students with disabilities or Specific Learning Disorders (SLD) are invited to directly inform the professor in charge of the course about the special arrangements for the exam that have been agreed with the Special Needs Unit. The professor has to be informed at least one week before the beginning of the examination session in order to provide students with the most suitable arrangements for each specific type of exam.