Last update:  09-Mar-2024

Course Organization:

•  Evaluation :
•  Quiz                                                                                                                          : 5 * 6% = 30%
•  Project replication of the results of a scientific paper + critical analysis    : 70%    (groups of 4 students)
•  Documents :
•  Slides of the courses (see below in the course's schedule)

Outline of the course:

• Learning is about extrapolating predictions and/or regularities from limited data (even if sometimes there is an apparently very large data set) either in a stationary environment (classical induction) or in a changing one (domain adaptation, transfer learning).
  
  This brings questions that will be core topics of the course:
  
  
  
  1. In the case of stationary environments, there is a well developed theory called "the statistical theory of learning", but what can be done when the environment changes between the training data and the testing one?
  2. Learning may imply several learning agents as in ensemble learning: what information should they exchange?
  3. Learning may also imply several learning tasks, as in domain adaptation, transfer learning or online learning. Again, what should be exchanged between contexts?
  
  In this class, we will ask ourselves:
  
  
  • What is transferred in learning, and specially in non stationary environments?
  • How to measure the difficulty of learning an example?
  • What is the best way to organize a curriculum? Or what is a good teaching strategy

What you will acquire in the course:

1. You will get an understanding of the problem of induction. You will ponder about the kind of guarantees that we can try to obtain as well as to which assumptions should we make and then be aware of when claiming theorems. In order to do so, you will look at the standard statistical learning theory and examine extensions to it.


2. You will gain a better appreciation of what is involved in ensemble methods and, in particular, what kind of communication there can be between the agents and how their advices can be combined in order to get a decision.
3. You will know what are the problems that face "out of distribution" learning, and get knowledge about some recent directions of research on this problem.
4. As part of a project group of students, you will study in depth a recent research paper and try to replicate its results.
5. As part of a project group of students, you will exercice a critical apparaisal of a research paper.



---

Tentative schedule:

Dates :	Topics   (tentative schedule)	References, exercises and homeworks
11-01-2024 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) Learning as generalization    - The statistical theory of learning for a stationary world. (The In-Distribution assumption)    - Why it does not seem to apply to deep learning.
18-01-2024 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) When the distribution P_X is changed to better learn When the learning agent modifies the input distribution: Boosting, bagging, Random Forests. What they are. Theoretical approaches. Extension to other ensemble methods? The LUPI framework. Learning using a given input space, and being tested using an input space which is a subspace of the learning input space. Illustration with Early Classification of Time Series	• Quiz No 1 • Slides of the class
26-01-2024 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) No class!!
01-02-2024 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) When the training distribution differs from the test one Slides (1) What to do with imbalanced data classes Learning with positive only examples Semi-supervised learning Active learning Learning agents that communicate Slides (2) Co-training. Having independent and complementary views. A curiosity: blending. Distillation. Two agents: one acting as a teacher, the other as a student. Modification of the training examples. Points towards curriculum learning. Multi-task learning. Minimizing the differences between the learnt hypotheses. The MDLp (Minimum Description Length Problem). Communication between “agents”. Application to analogy making.	• Quiz No 2
08-02-2024 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) When P(X) is changed but not P(Y|X) Slides of the class Covariate shift / Domain Adaptation. An intriguing idea: tracking (Communicating agents)	• Quiz No 3
15-02-2024 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) When P(Y|X) changes (1) Slides of the class Continual learning. Transfer learning. - What are the ingredients for a successful (detrimental) transfer? - What role for the source? Role and measure of similarity between source and target in TL? Sequencing effects and Curriculum learning.      - A geometry on the space of learning tasks	• Quiz No 4 (ICML-2021)-An Information-Geometric Distance on the Space of Tasks. (ICML-2021)-Continual Learning in the Teacher-Student Setup: Impact of Task Similarity.
29-02-2024 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) When P(Y|X) changes (2) Slides of the class Measuring the difficulty of new examples Catastrophic forgetting: explanations and ways to mitigate it The case of online learning (a succession of small transfers) Few shot learning (if time allows it)	• Quiz No 5

 

References and web sites:

•   Barra V. & Cornuéjols A. & Miclet L.: Apprentissage artificiel. Concept et algorithmes. De Bayes et Hume au Deep Learning. Eyrolles, 2021 (4ème édition)
    (A very comprehensive book on Machine Learning. In French.)

Books on Machine Learning in general:

•  Barber D. : Bayesian Learning and Machine Learning. Cambridge University Press, 2012.
   (A Bayesian perspective on Machine Learning.)
•  Duda, Hart & Stork : Pattern classification (2nd éd.). Wiley-Interscience, 2001.
   (Oriented towards Pattern Recognition. Very good graphics. Still a reference.)
•  Flach P. : Machine Learning. The art and science of algorithms that make sense of data. Cambridge University Press, 2012.
   (A good introductory book on Machine Learning. Quick on some subjects and does not really cover data mining.)
•  Hand, Mannila & Smyth : Data mining. Springer, 2001.
   (Livre assez complet, mais nécessairement plus superficiel car couvrant aussi la fouille de données)
•  Hastie, Tibshirani & Friedman : The elements of statistical learning. Data mining, inference and prediction. Springer, (2nd Ed. 2009).
   (A bit demanding, but very informative. The form is outstanding.)
•  Haykin S. : Neural netwoks. A comprehensive foundation. Prentice Hall, 1999.
   (An incredible source of information)
•  Haykin S. : Neural netwoks and Learning Machines. Prentice Hall, 2008.
   (Different from the 1999 edition, and yet still remarkable, and worth the reading.)
•  Mitchell T. : Machine Learning. McGraw Hill, 1997.
    (A good introductory book, but outdated on many subjects.)
•  Marsland S. : Machine Learning. An algorithmic perspective. CRC Press, 2009.
    (An introductory book, rather superficial, with bits of code in Python.)
•  Murphy S. : Machine Learning. A probabilistic perspective. MIT Press, 2012.
    (The title says it all. A book incredibly comprehensive from a Bayesian perspective.)

Books on specific topics:

•  Bishop C. : Neural networks for pattern recognition. Clarendon Press, 1995.
   (A good introductory book. Oriented towards connectionism but discusses many important issues in Machine Learning as well.)
•  Cristianini N. & Shawe-Taylor J. : Support Vectors Machines and other kernel-based learning methods. Cambridge University Press, 2000.
•  Webb A. : Statistical pattern recognition. Wiley, (3rd. Ed., 2011).
   (A good introductory book that deserves to be better known.)
  

---

Guidelines for the projects:

This year, your assignment in the projects is to choose an article from the list provided below, understand it, and, most importantly, try to replicate the experiments.
Rules are as follows:
• Projects are conducted by team of 4 students. Team members are responsible for decompose the work so that all members contribute meaningfully to the project.
The following deliverables are expected:
1. Before ( January 18th 2024 ), the choice of the paper and the names of the members of the team.
2. ( February, 29th 2024 ) Final report : 10 pages (strict. Reports longer than 10 pages will not be read!)
The interlediary report and the final reports must be turned down in the ICML (Int. Conf. on Machine Learning) format for papers. The deliverables will be evaluated taking into account:
• The clarity of the analysis of the chosen article. The paper should briefly present, in no more than two to four pages (out of the 10 allotted), the problem studied in the article, the main ideas and statements presented in the paper. The remaining pages will present the experiments carried out by the team, the possible difficulties, the results obtained and a comparison with the results presented by the authors of the article used as a basis for the project.


• The rigor and the extensive character of the analysis and/or the experiments carried out. A project that really answers the questions and possible doubts of the reviewers on the interest of the method and on the announced performances will obtain a higher score.


• Expression, clarity of explanation and quality of exposition. Reports can be written in French or English, as long as they are clear and well written.

---

List of the papers to chose from (all recent ones)


1. Taken (2020) In search of lost domain generalization Cited 823 times!
2. (ICML-2023) In Search for a Generalizable Method for Source Free Domain Adaptation.
3. Taken (ICML-2023) Domain Adaptation for Time Series Under Feature and Label Shifts.
4. Taken (ICML-2023) Taxonomy-Structured Domain Adaptation.
5. Taken (ICML-2023) Random Teachers are Good Teachers.
6. Taken (ICML-Workshop-2023) Studying Generalization on Memory-Based Methods in Continual Learning.
7. Taken (AAAI-2022) Being Friends Instead of Adversaries- Deep Networks Learn from Data Simplified by Other Networks.
8. (Neurips-2022) Does knowledge distillation really works?.
9. Taken (ICML-2021) Continual learning in the teacher-student setup: impact of task similarity.
10. (ICML-2020) Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.
11. Taken (ICCV-2021) Deep co-training with task decomposition for semi-supervised domain adaptation.
12. Taken (ICML-2022) Head2toe: Utilizing intermediate representations for better transfer learning.
13. Taken (ICLR-2022) Fortuitous forgetting in connectionist networks.

---

Projects chosen by the students:

Names :	Projects :
Dorin DONCENCO Ayoub HAMMAL Quentin LETELLIER Benjamin MAUDET	(ICML-2022) Head2toe: Utilizing intermediate representations for better transfer learning. (final report: YES )
Shah Fahad HOSSAIN Thanh Gia Hieu KHUONG Alex-Razvan ISPAS Benedictus Kent RACHMAT Diego Andrès TORRES GUARIN	(AAAI-2022) Being Friends Instead of Adversaries- Deep Networks Learn from Data Simplified by Other Networks. (final report: YES )
Hala ALBAHLOUL Gabriel BECQUET Jacobo RUIZ Bias SHI	(ICLR-2022) Fortuitous forgetting in connectionist networks. (final report: YES )
Arthur BABIN Rokaya BENAZZI Clément DILASSER Nicolas RICHET	(2020) In search of lost domain generalization. (final report: YES )
Nathan Maligeay Leonardo Masci Hugo Pointier Mahdiehsadat Seyedian	(ICCV-2021) Deep co-training with task decomposition for semi-supervised domain adaptation. (final report: YES )
Ambroise BERTIN Maelle CORNEC Jules DUPONT Suzanne GUILTEAUX	(ICML-2023) Random Teachers are Good Teachers. (final report: YES )
Sina Khanagha Ignacio Lopez Alberto Magina Anooga Suresh	(ICML-Workshop-2023) Studying Generalization on Memory-Based Methods in Continual Learning. (final report: YES )
Julian AGUDELO Gabriel CLEREMPUY Mathieu GUIMONT lauriane MOUSSET Anooga SURESH	(ICML-2021) Continual learning in the teacher-student setup: impact of task similarity. (final report: NO )
Julien GIOVANAZZI Céline GUO Fiona HAK Kamenan N'GADI	(ICML-2023) Domain Adaptation for Time Series Under Feature and Label Shifts. (final report: YES )
Malek BAROUDI Nada KASSARA Afonso PONCE Matthieu VERLYNDE	(ICML-2023) Taxonomy-Structured Domain Adaptation. (final report: YES )
Marine TOGNIA-TONOU	(ICML-2020) Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. (final report: YES )

Propositions for internships :

 

sujet	Institution
sujet	Institution
sujet	Institution
sujet	Institution
sujet	Institution