Last update:  16-January-2025

Course Organization:

•  Evaluation :
•  Quiz                                                                                                                          : 5 * 6% = 30%
•  Project replication of the results of a scientific paper + critical analysis    : 70%    (groups of 4 students)
•  Documents :
•  Slides of the courses (see below in the course's schedule)

Outline of the course:

• Learning is about extrapolating predictions and/or regularities from limited data (even if sometimes there is an apparently very large data set) either in a stationary environment (classical induction) or in a changing one (domain adaptation, transfer learning).
  
  This brings questions that will be core topics of the course:
  
  
  
  1. In the case of stationary environments, there is a well developed theory called "the statistical theory of learning", but what can be done when the environment changes between the training data and the testing one?
  2. Learning may imply several learning agents as in ensemble learning: what information should they exchange?
  3. Learning may also imply several learning tasks, as in domain adaptation, transfer learning or online learning. Again, what should be exchanged between contexts?
  
  In this class, we will ask ourselves:
  
  
  • What is transferred in learning, and specially in non stationary environments?
  • How to measure the difficulty of learning an example?
  • What is the best way to organize a curriculum? Or what is a good teaching strategy

What you will acquire in the course:

1. You will get an understanding of the problem of induction. You will ponder about the kind of guarantees that we can try to obtain as well as to which assumptions should we make and then be aware of when claiming theorems. In order to do so, you will look at the standard statistical learning theory and examine extensions to it.


2. You will gain a better appreciation of what is involved in ensemble methods and, in particular, what kind of communication there can be between the agents and how their advices can be combined in order to get a decision.
3. You will know what are the problems that face "out of distribution" learning, and get knowledge about some recent directions of research on this problem.
4. As part of a project group of students, you will study in depth a recent research paper and try to replicate its results.
5. As part of a project group of students, you will exercice a critical apparaisal of a research paper.



---

Tentative schedule:

Dates :	Topics   (tentative schedule)	References, exercises and homeworks
09-01-2025 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols)    - Introduction to the course. Out-of-Distribution learning. Central question of the course: What makes you believe that what you have learned so far from (source) training examples or task will be relevant to new (target) examples or tasks? Examination of classical induction    - The statistical theory of learning for a stationary world. (The In-Distribution assumption)
16-01-2025 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) Out-of-Distribution learning for the sake of learning better or with less (annotated) examples Semi-supervised learning Learning when classes are severely unbalanced Learning from positive examples only Active learning O.O.D. through multiple agents Co-training, self-training and tri-training Boosting (after a reminder on SVMs) Blending Domain adaptation and tracking Domain adaptation (co-variate shift) Tracking: an intriguing idea. Domain adaptation without domain change! Extension to other ensemble methods?	• Quiz No 1 •
23-01-2025 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) Transfer learning What can be transferred: Representation Underlying regularities supposedly more stable among learning tasks The decision function Others: the loss function, the optimizer, ... (1) Learning a representation Fine tuning Self-supervised learning Contrastive learning Foundation models. Prompting. Zero-shot learning (2) Discovering underlying regularities to transfer Domain adversarial learning Invariant Risk Minimization	• Quiz No 2 • Slides of the class
30-01-2025 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) No class!!
06-02-2025 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) Transfer learning (3) Transfering the decision function instead of the representation TransBoost - What are the ingredients for a successful (detrimental) transfer? - What role for the source? Role and measure of similarity between source and target in TL? Analogy making and ordering effects. The MDLp (Minimum Description Length Problem) Cognitive tunnel effect - How to transfer from one conceptual domain to another one. Is there a better ordering? Learning from a teacher Slides of the class LUPI (Learning Using Privileged Information): not the same input space X between teacher and learner. Distillation. Not the same H. Prompting a foundation model.	• Quiz No 3 CORNUÉJOLS, Antoine. Some thoughts about transfer learning. What role for the source domain?. International Journal of Approximate Reasoning, 2024, vol. 166, p. 109107. Gou, J. and Yu, B. and Maybank, Stephen and Tao, Da. (2021) Knowledge distillation: a survey. International Journal of Computer Vision 129, pp.1789-1819. PHUONG, Mary et LAMPERT, Christoph. Towards understanding knowledge distillation. In : International conference on machine learning. PMLR, 2019. p. 5142-5151.
13-02-2025 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) Curriculum learning Sequence of tasks. How to organize them? How to measure the difficulty of new examples Catastrophic forgetting. Coaching. Not the same gain function. Curriculum: the space of tasks and the space of biases.	• Quiz No 4 (ICML-2021)-An Information-Geometric Distance on the Space of Tasks. (ICML-2021)-Continual Learning in the Teacher-Student Setup: Impact of Task Similarity.
20-02-2025 09h00 - 12h15 (Salle B-107)	(Antoine Cornuéjols) On-line learning. A succession of small transfer steps Slides of the class A theoretical perspective. Can we still guarantee a performance level even against an adversary who controls the target concept and can change it arbitrarily? Empirical approaches. General conclusions	• Quiz No 5 • Final reports on the projects are due

 

References and web sites:

•   Barra V. & Cornuéjols A. & Miclet L.: Apprentissage artificiel. Concept et algorithmes. De Bayes et Hume au Deep Learning. Eyrolles, 2021 (4ème édition)
    (A very comprehensive book on Machine Learning. In French.)
•   Torralba A. & Isola Ph. & Freeman W.: Foundations of Computer Vision. MIT Press, 2024.
    (Speaks mainly of computer vison obviously. But an incredible synthesis of learning problems and methods.
    Highly recommended.)

Books on Machine Learning in general:

•  Barber D. : Bayesian Learning and Machine Learning. Cambridge University Press, 2012.
   (A Bayesian perspective on Machine Learning.)
•  Bishop Ch. : Deep Learning. Foundations and Concepts. MIT Press, 2024.
   (Quite pedagogical.)
•  Duda, Hart & Stork : Pattern classification (2nd éd.). Wiley-Interscience, 2001.
   (Oriented towards Pattern Recognition. Very good graphics. Still a reference.)
•  Flach P. : Machine Learning. The art and science of algorithms that make sense of data. Cambridge University Press, 2012.
   (A good introductory book on Machine Learning. Quick on some subjects and does not really cover data mining. But a very original and often quite illuminating perspective.)
•  Hastie, Tibshirani & Friedman : The elements of statistical learning. Data mining, inference and prediction. Springer, (2nd Ed. 2009).
   (A bit demanding, but very informative. The form is outstanding.)
•  Haykin S. : Neural netwoks. A comprehensive foundation. Prentice Hall, 1999.
   (An incredible source of information)
•  Haykin S. : Neural netwoks and Learning Machines. Prentice Hall, 2008.
   (Different from the 1999 edition, and yet still remarkable, and worth the reading.)
•  Murphy K. : Machine Learning. A probabilistic perspective. MIT Press, 2012.
    (The title says it all. A book incredibly comprehensive from a Bayesian perspective.)
•  Murphy K. : Probabilistic Machine Learning. A introduction. MIT Press, 2022.
    (The new edition, first of two volumes.)
•  Murphy K. : Probabilistic Machine Learning. Advanced topics. MIT Press, 2022.
    (The new edition, second of two volumes. Lots of recent works. A little bit of a patchwork.)
•  Sejnowski T. : The deep learning revolution. MIT Press, 2018.
    (Provides a personal historical perspective. By someone who should have shared the 2024 Nobel Prize.)
•  Zou Zhi-Hua. : Machine Learning. Springer, 2016.
    (A very good introductory book)

Books on specific topics:

•  Bishop C. : Neural networks for pattern recognition. Clarendon Press, 1995.
   (A good introductory book. Oriented towards connectionism but discusses many important issues in Machine Learning as well.)
•  Cristianini N. & Shawe-Taylor J. : Support Vectors Machines and other kernel-based learning methods. Cambridge University Press, 2000.
•  Webb A. : Statistical pattern recognition. Wiley, (3rd. Ed., 2011).
   (A good introductory book that deserves to be better known.)
  

---

Guidelines for the projects:

This year, your assignment in the projects is to choose an article from the list provided below, understand it, and, most importantly, try to replicate the experiments.

Rules are as follows:
• Projects are conducted by team of 4 students. Team members are responsible for decompose the work so that all members contribute meaningfully to the project.

The following deliverables are expected:
1. Before ( January 16th 2025 ), the choice of the paper and the names of the members of the team.
2. ( February, 20th 2025 ) Final report : 10 pages (strict. Reports longer than 10 pages will not be read!)

The intermediary report and the final report must be turned down in the ICML (Int. Conf. on Machine Learning) format for papers.

The deliverables will be evaluated taking into account:


• The clarity of the analysis of the chosen article. The paper should briefly present, in no more than two to four pages (out of the 10 allotted), the problem studied in the article, the main ideas and statements presented in the paper. The remaining pages will present the experiments carried out by the team, the possible difficulties, the results obtained and a comparison with the results presented by the authors of the article used as a basis for the project.


• The rigor and the extensive character of the analysis and/or the experiments carried out. A project that really answers the questions and possible doubts of the reviewers on the interest of the method and on the announced performances will obtain a higher score.


• Expression, clarity of explanation and quality of exposition. Reports can be written in French or English, as long as they are clear and well written.



---

List of the papers to chose from (all recent ones)


1. Taken (2020) In search of lost domain generalization.
2. Taken (ICLR-2024) Self-Supervised Dataset Distillation for Transfer Learning.
3. New (ICLR-2024) Transferring Learning Trajectories of Neural Networks.
4. (ICML-2023) Taxonomy-Structured Domain Adaptation.
5. Taken (ICML-2023) Random Teachers are Good Teachers.
6. (ICML-Workshop-2023) Studying Generalization on Memory-Based Methods in Continual Learning.
7. New (ICLR-2024) Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP.
8. Taken (Neurips-2022) Does knowledge distillation really works?.
9. Taken (ICML-2021) Continual learning in the teacher-student setup: impact of task similarity.
10. Partly taken (ICML-2020) Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.
11. (ICCV-2021) Deep co-training with task decomposition for semi-supervised domain adaptation.
12. Taken (ICML-2024) Transferring Knowledge From Large Foundation Models to Small Downstream Models.

13. Partly taken

    (ICML-2024) Pseudo-Calibration-Improving Predictive Uncertainty Estimation in Unsupervised Domain Adaptation.
14. Taken (ICML-2024) Soft_Contrastive_Learning.
15. Taken (ICML-2024) Let Go of Your Labels with Unsupervised Transfer.
16. New (IJCAI-2024) Alleviating Imbalanced Pseudo-label Distribution- Self-Supervised Multi-Source Domain Adaptation with Label-specific Confidence.
17. New (IJCAI-2024) Reconfgurability-Aware Selection for Contrastive Active Domain Adaptation.
18. Taken (ICLR-2022) Fortuitous forgetting in connectionist networks.

---

Projects chosen by the students:

Names :	Projects :
Student-1 Student-2 Student-3 Student-4	(Conference) Paper title. (final report: YES )
Biruk Abere Ambaw Emirhan Bilgiç Amruth Srivathsan Aaron Weissberg	(Neurips-2022) Does knowledge distillation really works?. (final report: NO )
Ying Lai Huichi Kuo Niloofar Sahebalzamany Student-4	(ICML-2023) Random Teachers are Good Teachers. (final report: NO )
Titouan Chambe Maiwen Demeulle Antoine Loth Titouan Chambe	(ICML-2024) Transferring Knowledge From Large Foundation Models to Small Downstream Models. (final report: NO )
Adrien Bouchet Vincent Fourmigué Elise Roth Timothée Sanchez	(ICML-2024) Soft_Contrastive_Learning. (final report: NO )
Lisa Achard Manon Chartrin Sovanleng Ly Nabil Meridja	(ICML-2021) Continual learning in the teacher-student setup: impact of task similarity. (final report: NO )
Linh Dinh Mélodie Fleury Nicolas Geffroy Matéo Petitet	(ICLR-2022) Fortuitous forgetting in connectionist networks. (final report: NO )
Camille De Amorim Mapathe Faye Jaffar Gura Student-4	(2020) In search of lost domain generalization. (final report: NO )
Jose Espinosa Martin Masson Dana Zlotenko Student-4	(ICLR-2024) Self-Supervised Dataset Distillation for Transfer Learning. (final report: NO )
Raphaël Rubrice Student-2 Student-3 Student-4	(ICML-2024) Pseudo-Calibration-Improving Predictive Uncertainty Estimation in Unsupervised Domain Adaptation. (final report: NO )
Noémie Bozier Nathan Carré Anakim Gualdoni Georges Hanna	(ICML-2024) Let Go of Your Labels with Unsupervised Transfer. (final report: NO )
Younes Djemmal Student-2 Student-3 Student-4	(ICML-2020) Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. (final report: NO )