At the end of the course the student will: - Know the key concepts of Information Theory - Understand the meaning of Entropy for Data Science - Be aware of the application of Information Theory to Classifiers - Be aware of the application of Information Theory to Data Compression and Representation - Recognize the role of Information Theory for Cryptography and some of the most important Crypto Algorithms - Know how to realize a Matlab of Python project to analyze an Information Theory problem

Very basic programming skills in Matlab or Python.

The course topics are: 1. Introduction to Information Theory. Application to Classifiers (20 h) 2. Application of Information Theory to Data Representation and Compression 1 (20 h) 3. Application of Information Theory to Data Representation and Compression 2 (20 h) 4. Information Theory and Cryptography. Algorithms for Cryptography (20 h)

The course is organized into four sections. For each section we propose an assignment that must be solved by writing a Matlab or Python program and a report. About half of the hours are dedicated to lectures (that will be recorded) and half to tutoring, where the teachers are available to help students with the assignment solution.

- Thomas M. Cover and Joy A. Thomas “Elements of Information Theory” - Aston Zhang, Zack C. Lipton, Mu Li, Alex J. Smola “Dive into Deep Learning” Chapter 18 “Information Theory” (available online) - Lecture recordings from last year (or new if required by the subject). - Notes from the teachers.

Lecture slides; Video lectures (previous years);

Exam: Written test; Optional oral exam; Group project;

The purpose of the exam for the Information Theory for Data Science course is to assess students' understanding of key concepts of information theory and their ability to apply them in real-world data science scenarios. The purpose of the exam is to verify that the student: - has acquired the basis of information theory, like the notions of entropy, information gain, Kullback–Leibler divergence, data compression, data cryptography and the link between them and Data Science; - is able to understand an information theory problem, model it, write a program to evaluate its performance in Matlab/Python, and critically discuss it in a professional technical report. There are two examination modes: exam A and exam B Description of Exam A (assignments + written exam of type A): The course is divided into 4 sections. For each section an assignment (to be solved in groups of up to 4 students) is proposed. The groups must deliver: - a pdf presentation (generated with ppt, beamer, ecc.) containing all the results, the figures and the required answers - all the Matlab (preferably) or Python programs. Each assignment is evaluated and receives a mark (max = 30). Bonuses are available for early submission of the assignment. The following bonus structure applies based on the submission timing relative to the assignment date: If submitted within 2 weeks: a bonus of +2 points is awarded (maximum score = 32). If submitted within 3 weeks: a bonus of +1 point is awarded (maximum score = 31). If submitted within 4 weeks: no bonus is awarded (maximum score = 30). Submissions after 4 weeks will not be accepted. The average assignment grade is obtained as the average of the four assignment grades. The final grade is obtained as the weighted average between the average assignment grade (70%) and the grade of a written exam of type A (30%). The students who achieve a weighted average of at least 30.5 will receive a mark of 30 cum laude. Description of exam B (no assignments, written exam of type B): - Written exam of type B without assignments (maximum grade = 25). The student can take an optional oral exam, that may decrease or increase the grade, and may reach 30 cum laude if the oral exam is outstanding. The written exam of type A consists of four questions taken from a list of questions made of about 20 questions (time available = 1.5 hours, no books or notes, maximum grade = 32). The written exam of type B consists of four questions from a list of questions made of about 50 questions (time available = 1.5 hours, no books or notes, maximum mark = 25).

In addition to the message sent by the online system, students with disabilities or Specific Learning Disorders (SLD) are invited to directly inform the professor in charge of the course about the special arrangements for the exam that have been agreed with the Special Needs Unit. The professor has to be informed at least one week before the beginning of the examination session in order to provide students with the most suitable arrangements for each specific type of exam.