Course Master AI: Learning Theory and Advanced Machine Learning

---

Homepage

Program of the class

Tentative schedule of the classes

References

Articles to chose from

Projects

Internships

Last update:  03-February-2026

Course Organization:

•  Evaluation :
•  Quiz                                                                                                                          : 5 * 6% = 30%
•  Project replication of the results of a scientific paper + critical analysis    : 70%    (groups of 4 students)
•  Documents :
•  Slides of the courses (see below in the course's schedule)

---

Outline of the course:

1. Classical inductive learning: I.I.D.
   • Basic assumptions
   • What kinds of guarantees
   • Transductive learning
   • Types of distribution shifts and transfer learning (Out-of-Distribution (O.O.D.))


2. Using O.O.D. learning to help solve an I.I.D. learning task
   • Semi-supervised learning (passive approach)
   • Imbalanced classes (active approach)
   • Active learning (active approach)
   • LUPI (Learning Using Privileged Information)
   • Multi-agent O.O.D. methods in the I.I.D. scenario
     • Boosting
     • Co-learning
     • Blending
     • Cognitive Tunnel Effect


3. O.O.D. at testing time
   • Designed to stem learning: Adversarial examples
   • Co-variate shift (or domain adaptation)
   • A special case: Tracking


4. O.O.D. in transfer learning (and multi-tasks learning)

   Using the assumption that there is something in common between the source and the target, what can be transfered includes:

   • A representation
   • The decision function
   • Something invariant, for instance causality relationships
   • Others: the loss function, the optimizer, …


5. Analogy making


6. Learning with a teacher: distillation


7. Curriculum learning
   • A teacher
   • Interferences between tasks
     • Measuring the difficulty of new examples
     • Catastrophic forgetting
     • A geometry on the space of tasks
     • The geometry can be non Euclidean. The path itself (order of tasks) conveys information


8. Online learning
   • Can we still guarantee something if the teacher acts as an adversary?
   • General empirical approaches to online learning. Solving the plasticity-stability trade-off


9. Conclusion

---

What you will acquire in the course:

1. You will get an understanding of the problem of induction. You will ponder about the kind of guarantees that we can try to obtain as well as to which assumptions should we make and then be aware of when claiming theorems. In order to do so, you will look at the standard statistical learning theory and examine extensions to it.


2. You will gain a better appreciation of what is involved in ensemble methods and, in particular, what kind of communication there can be between the agents and how their advices can be combined in order to get a decision.
3. You will know what are the problems that face "out of distribution" learning, and get knowledge about some recent directions of research on this problem.
4. As part of a project group of students, you will study in depth a recent research paper and try to replicate its results.
5. As part of a project group of students, you will exercice a critical apparaisal of a research paper.



---

Tentative schedule:

Dates :	Topics   (tentative schedule)	References, exercises and homeworks
08-01-2026 09h00 - 12h15 (Salle B-108)	(Antoine Cornuéjols)    - Introduction to the course. Out-of-Distribution learning. Central question of the course: What makes you believe that what has been learned so far from (source) training examples or task will be relevant to new (target) examples or tasks? Classical inductive learning: I.I.D. Basic assumptions What kinds of guarantees Transductive learning
15-01-2026 09h00 - 12h15 (Room C1.2.04 (AgroParisTech))	(Antoine Cornuéjols) Types of distribution shifts and transfer learning (Out-of-Distribution (O.O.D.)) Using O.O.D. learning to help solve an I.I.D. learning task Semi-supervised learning (passive approach) Imbalanced classes (active approach) Active learning (active approach) LUPI (Learning Using Privileged Information) Multi-agent O.O.D. methods in the I.I.D. scenario Co-learning Boosting Blending	• Quiz No 1
22-01-2026 09h00 - 12h15 (Room B-108 (P.U.I.O.))	(Antoine Cornuéjols) Using O.O.D. learning to help solve an I.I.D. learning task Multi-agent O.O.D. methods in the I.I.D. scenario Boosting (follow up) Blending O.O.D. at testing time Designed to stem learning: Adversarial examples Co-variate shift (or domain adaptation) A special case: Tracking O.O.D. in transfer learning Learning a representation for multiple tasks	• Quiz No 2
29-01-2026 09h00 - 12h15 (Room C1.2.01 (AgroParisTech))	(Antoine Cornuéjols) O.O.D. in transfer learning The classical view on transfer learning with neural networks How transferable are features in deep neural networks? Are the features learned during pretraining of foundational models general enough to enable fine tuning on any task? What does a foundation model know and how does it know it?	• Quiz No 3
05-02-2026 09h00 - 12h15 (Room C1.3.05 (AgroParisTech))	(Antoine Cornuéjols) O.O.D. in transfer learning Transfering the decision function Searching for invariances between domains	• Quiz No 4 CORNUÉJOLS, Antoine. Some thoughts about transfer learning. What role for the source domain?. International Journal of Approximate Reasoning, 2024, vol. 166, p. 109107.
12-02-2026 09h00 - 12h15 (Room C1.3.05 (AgroParisTech))	(Antoine Cornuéjols)	• Quiz No 5 Gou, J. and Yu, B. and Maybank, Stephen and Tao, Da. (2021) Knowledge distillation: a survey. International Journal of Computer Vision 129, pp.1789-1819. PHUONG, Mary et LAMPERT, Christoph. Towards understanding knowledge distillation. In : International conference on machine learning. PMLR, 2019. p. 5142-5151. (ICML-2021)-An Information-Geometric Distance on the Space of Tasks. (ICML-2021)-Continual Learning in the Teacher-Student Setup: Impact of Task Similarity.
19-02-2026 09h00 - 12h15 (Salle B-108)	(Antoine Cornuéjols)	• Final reports. Due date: February, 19th, 2026

 

References and web sites:

•   Barra V. & Cornuéjols A. & Miclet L.: Apprentissage artificiel. Concept et algorithmes. De Bayes et Hume au Deep Learning. Eyrolles, 2021 (4ème édition)
    (A very comprehensive book on Machine Learning. In French.)
•   Torralba A. & Isola Ph. & Freeman W.: Foundations of Computer Vision. MIT Press, 2024.
    (Speaks mainly of computer vison obviously. But an incredible synthesis of learning problems and methods.
    Highly recommended.)

Books on Machine Learning in general:

•  Barber D. : Bayesian Learning and Machine Learning. Cambridge University Press, 2012.
   (A Bayesian perspective on Machine Learning.)


•  Bishop Ch. : Deep Learning. Foundations and Concepts. MIT Press, 2024.
   (Quite pedagogical.)


•  Duda, Hart & Stork : Pattern classification (2nd éd.). Wiley-Interscience, 2001.
   (Oriented towards Pattern Recognition. Very good graphics. Still a reference.)


•  Flach P. : Machine Learning. The art and science of algorithms that make sense of data. Cambridge University Press, 2012.
   (A good introductory book on Machine Learning. Quick on some subjects and does not really cover data mining. But a very original and often quite illuminating perspective.)


•  Hastie, Tibshirani & Friedman : The elements of statistical learning. Data mining, inference and prediction. Springer, (2nd Ed. 2009).
   (A bit demanding, but very informative. The form is outstanding.)


•  Haykin S. : Neural netwoks. A comprehensive foundation. Prentice Hall, 1999.
   (An incredible source of information)


•  Haykin S. : Neural netwoks and Learning Machines. Prentice Hall, 2008.
   (Different from the 1999 edition, and yet still remarkable, and worth the reading.)
•  Murphy K. : Machine Learning. A probabilistic perspective. MIT Press, 2012.
    (The title says it all. A book incredibly comprehensive from a Bayesian perspective.)


•  Murphy K. : Probabilistic Machine Learning. A introduction. MIT Press, 2022.
    (The new edition, first of two volumes.)


•  Murphy K. : Probabilistic Machine Learning. Advanced topics. MIT Press, 2022.
    (The new edition, second of two volumes. Lots of recent works. A little bit of a patchwork.)


•  Sejnowski T. : The deep learning revolution. MIT Press, 2018.
    (Provides a personal historical perspective. By someone who should have shared the 2024 Nobel Prize.)


•  Zou Zhi-Hua. : Machine Learning. Springer, 2016.
    (A very good introductory book)

Books on specific topics:

•  Bishop C. : Neural networks for pattern recognition. Clarendon Press, 1995.
   (A good introductory book. Oriented towards connectionism but discusses many important issues in Machine Learning as well.)


•  Cristianini N. & Shawe-Taylor J. : Support Vectors Machines and other kernel-based learning methods. Cambridge University Press, 2000.


•  Webb A. : Statistical pattern recognition. Wiley, (3rd. Ed., 2011).
   (A good introductory book that deserves to be better known.)
  

---

Guidelines for the projects:

This year, your assignment in the projects is to choose an article from the list provided below, understand it, and, most importantly, try to replicate the experiments.

Rules are as follows:
• Projects are conducted by team of 4 students. Team members are responsible for decompose the work so that all members contribute meaningfully to the project.

The following deliverables are expected:
1. Before ( January 15th 2026 ), the choice of the paper and the names of the members of the team.
2. ( February, 19th 2026 ) Final report : 10 pages (strict. Reports longer than 10 pages will not be read!)

The intermediary report and the final report must be turned down in the ICML (Int. Conf. on Machine Learning) format for papers.

The deliverables will be evaluated taking into account:


• The clarity of the analysis of the chosen article. The paper should briefly present, in no more than two to four pages (out of the 10 allotted), the problem studied in the article, the main ideas and statements presented in the paper. The remaining pages will present the experiments carried out by the team, the possible difficulties, the results obtained and a comparison with the results presented by the authors of the article used as a basis for the project.


• The rigor and the extensive character of the analysis and/or the experiments carried out. A project that really answers the questions and possible doubts of the reviewers on the interest of the method and on the announced performances will obtain a higher score.


• Expression, clarity of explanation and quality of exposition. Reports can be written in French or English, as long as they are clear and well written.



---

List of the papers to chose from


1. Taken (2025) These Are Not All the Features You Are Looking For- A Fundamental Bottleneck in Supervised Pretraining.


2. Taken (2022) Emergent world representations- Exploring a sequence model trained on a synthetic task. (Playing Othello with LLMs)


3. Taken (2022) Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models. (and the blog entry, even more detailed.) (Playing Chess with LLMs)


4. Taken (AAI-2025) Tabular out-of-distribution data synthesis for enhancing robustness. (Experiments without deep learning. Interesting nonetheless.)


5. Taken (AAAI-2025) RUNA- Object-level Out-of-Distribution Detection via Regional Uncertainty Alignment of Multimodal Representations..


6. (AAAI-2025)-Progressive Self-Learning for Domain Adaptation on Symbolic Regression of Integer Sequences..


7. Taken (2020) In search of lost domain generalization.


8. (ICLR-2024) Self-Supervised Dataset Distillation for Transfer Learning. (Powerful computational ressources required)
9. Taken (ICLR-2024) Transferring Learning Trajectories of Neural Networks.
10. Taken (ICML-2023) Taxonomy-Structured Domain Adaptation.
11. Taken (ICML-2023) Random Teachers are Good Teachers.
12. (ICML-Workshop-2023) Studying Generalization on Memory-Based Methods in Continual Learning.
13. (ICLR-2024) Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP.
14. Taken (Neurips-2022) Does knowledge distillation really works?.
15. (ICML-2021) Continual learning in the teacher-student setup: impact of task similarity.
16. (ICML-2020) Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. (Powerful computational ressources required)
17. (ICCV-2021) Deep co-training with task decomposition for semi-supervised domain adaptation.
18. Taken (ICML-2024) Transferring Knowledge From Large Foundation Models to Small Downstream Models.

19. (ICML-2024) Pseudo-Calibration-Improving Predictive Uncertainty Estimation in Unsupervised Domain Adaptation. (Powerful computational ressources required)
20. Taken (ICML-2024) Let Go of Your Labels with Unsupervised Transfer.
21. (IJCAI-2024) Alleviating Imbalanced Pseudo-label Distribution- Self-Supervised Multi-Source Domain Adaptation with Label-specific Confidence.

---

Projects chosen by the students:

Names :	Projects :
Kseniia PAVLOVA Elizaveta SIROTINA Polina SOLOVEVA Ekaterina TIMOFEEVA	(2022) Emergent world representations- Exploring a sequence model trained on a synthetic task. (final report: NO )
Zhetessov CAREER A. ZHAMANKHAN Menghor THUO	(ICML-2024) Transferring Knowledge From Large Foundation Models to Small Downstream Models. (final report: NO )
Tristan BOUTEVILLE Tom GORTANA Nicolas LENGLET Donatien WALLAERT	(2022) Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models. (final report: NO )
Feddy IMMOULA Tyler MARINO Thibault POUX	(AAI-2025) Tabular out-of-distribution data synthesis for enhancing robustness. (final report: NO )
Darpan ASWAL Marija BRKIC Mina GORANOVIC	(ICML-2023) Random Teachers are Good Teachers. (final report: NO )
Rasul ALAKBARLI Mahammad NURIYEV Petko PETKO Dmitrije ZDRALE	(Neurips-2022) Does knowledge distillation really works?. (final report: NO )
Marius CHARLES Mafalda FRERE Mathis RESCANIERES	(2020) In search of lost domain generalization. (final report: NO )
Mateus Fereira de Oliveira Chavez Damien Glaizal Inès Sauvanaud Charlotte Tibi	(ICML-2024) Let Go of Your Labels with Unsupervised Transfer. (final report: NO )
Pierre-Jean GOUZE Baptiste GRANIER Valentine MICHELET	(AAAI-2025) RUNA- Object-level Out-of-Distribution Detection via Regional Uncertainty Alignment of Multimodal Representations.. (final report: NO )
Lamia Hossein Mohamed Idir Meradji	(2025) These Are Not All the Features You Are Looking For- A Fundamental Bottleneck in Supervised Pretraining. (final report: NO )
Marie MEIER Eliott SERRON	(ICML-2023) Taxonomy-Structured Domain Adaptation. (final report: NO )
René GARCIA	(ICLR-2024) Transferring Learning Trajectories of Neural Networks. (final report: NO )
Yohan BEAUMATIN François FARAMUS Alan TURBOT	(ICLR-2024) Transferring Learning Trajectories of Neural Networks. (final report: YES NO )
Student_1 Student_2 Student_3 Student_4	(Conference) Paper title. (final report: YES NO )




---

Propositions for internships :

 

sujet	Institution
sujet	Institution
sujet	Institution
sujet	Institution
sujet	Institution