Homepage

Last update:  August-2-2026

Course Organization:

•  Evaluation :
•  Quiz                                                                                                                          : 5 * 6% = 30%
•  Project replication of the results of a scientific paper + critical analysis    : 70%    (groups of 4 students)
•  Documents:
•  Slides of the courses (see below in the course's schedule)
•  Structure of each class:
• A quiz followed by the answers
• The course enlighting concepts related to the topic of the day
• A thought-provoking segment designed to broaden the mind: research questions and opening new perspectives

Outline of the course:

1. Classical inductive learning: I.I.D. (Independent and Identically Distributed data)
   
   
   • The goal: to learn a function that is valid under a hypothesis across the entire input space X
   • True risk and empirical risk
   • The necessity of bias
   • The possibility of a theory of induction: PAC learning and statistical learning theory
   • A fundamental reliance on the assumption of a stationary world.
   • The question of the scope of this theory for superparameterized models: LLMs and foundation models
   
   
   • Opening the mind:
   • Double descent and groking
   
   
2. Classical Inductive Learning (continued). When the distribution of training examples is not the same as the generalization distribution
   
   
   • Imbalanced classes (active approach)
   • Active learning (active approach)
   • Semi-supervised learning (passive approach)
   • Covariate shift (active approach)
   • Boosting (active approach)
   
   
   • Opening the mind:
   • Co-learning


3. A Context-Specific Perspective on Inductive Learning: transductive learning
   
   
   • What is transductive learning and why it should be more efficient that classical inductive learning
   • Attentional models. Transformers as a specific approach
   • Few-shot learning
   • Fine-tuning large models
   
   
   • Opening the mind:
   • A side view on transformers
   • Playing Othello and chess with LLMs. What it tells about representation learning


4. Transfer learning by a single agent

   Inside a single domain

   • Domain adaptation
   • Measuring the difficulty of new examples
   • Adversarial learning

   Between domains

   • Source and target domains
   • Transfer learning in neural networks
   • The Transboost method
   • What role for the source?

   A fundamental question: what is transfered between the source and the target?

   • Assuming something in common and constant
   • The representation: common embedding
   • Causality relationships
   • The decision function
   • Others: the loss function, the optimizer, ...
   • Assumpingh maximum relevance of the source for the target queries: minimizing the adaptation to solve the target query
   
   Opening the mind:
   • Analogical making and the MDLP (Minimum Description Length principle)
   • Cognitive blending
   • The cognitive tunnel effect


5. Transfer learning with multiple agents


• LUPI: Learning with Privileged Information
• Multimodal and multi task learning
• Learning with a teacher: distillation
• Learning with a teacher: curriculum learning


• Opening the mind:
• Fiber bundle space and curvature
• Sequencing effects in Human and Machine Learning
• The path itself (order of tasks) conveys information


6. Online learning
   
   
   • Tracking
   • Online learning against any distribution changes. Can we still guarantee something if the teacher acts as an adversary?
   • Catastrophic forgetting
   • General empirical approaches to online learning. Solving the plasticity-stability trade-off
   
   
   • Opening the mind:
   • How to measure sequencing effects in LLMs?


7. Conclusion

---

What you will acquire in the course:

1. You will get a deeper understanding of the problem of induction. You will ponder about the kind of guarantees that we can try to obtain as well as to which assumptions should we make and then be aware of when claiming theorems. In order to do so, you will look at the standard statistical learning theory and examine extensions to it. This will prepare you to understand the challenges of out-of-distribution (O.O.D.) learning
2. You will know what are the problems that face "out of distribution" learning, and get knowledge about some recent directions of research on this problem.
3. As part of a project group of students, you will study in depth a recent research paper and try to replicate its results.
4. As part of a project group of students, you will exercice a critical apparaisal of a research paper.



---

Tentative schedule:

Dates :	Topics   (tentative schedule)	References, exercises and homeworks
08-01-2026 09h00 - 12h15 (Room C1.2.04 (AgroParisTech))	(Antoine Cornuéjols)    - Introduction to the course. Out-of-Distribution learning. Central question of the course: What makes you believe that what has been learned so far from (source) training examples or task will be relevant to new (target) examples or tasks? Classical inductive learning: I.I.D. Basic assumptions What kinds of guarantees Transductive learning
15-01-2026 09h00 - 12h15 (Room C1.2.04 (AgroParisTech))	(Antoine Cornuéjols)	• Quiz No 1
22-01-2026 09h00 - 12h15 (Room C1.2.01 (AgroParisTech))	(Antoine Cornuéjols)	• Quiz No 2
29-01-2026 09h00 - 12h15 (Room C1.2.01 (AgroParisTech))	(Antoine Cornuéjols)	• Quiz No 3
05-02-2026 09h00 - 12h15 (Room C1.3.05 (AgroParisTech))	(Antoine Cornuéjols)	• Quiz No 4
12-02-2026 09h00 - 12h15 (Room C1.3.01 (AgroParisTech))	(Antoine Cornuéjols)	• Quiz No 5
19-02-2026 09h00 - 12h15 (Room C1.3.01 (AgroParisTech))	(Antoine Cornuéjols)	• Final reports. Due date: February, XXX, 2027

 

References and web sites:

•   Barra V. & Cornuéjols A. & Miclet L.: Apprentissage artificiel. Concept et algorithmes. De Bayes et Hume au Deep Learning. Eyrolles, 2021 (4ème édition)
    (A very comprehensive book on Machine Learning. In French.)
•   Torralba A. & Isola Ph. & Freeman W.: Foundations of Computer Vision. MIT Press, 2024.
    (Speaks mainly of computer vison obviously. But an incredible synthesis of learning problems and methods.
    Highly recommended.)

Books on Machine Learning in general:

•  Barber D. : Bayesian Learning and Machine Learning. Cambridge University Press, 2012.
   (A Bayesian perspective on Machine Learning.)


•  Bishop Ch. : Deep Learning. Foundations and Concepts. MIT Press, 2024.
   (Quite pedagogical.)


•  Duda, Hart & Stork : Pattern classification (2nd éd.). Wiley-Interscience, 2001.
   (Oriented towards Pattern Recognition. Very good graphics. Still a reference.)


•  Flach P. : Machine Learning. The art and science of algorithms that make sense of data. Cambridge University Press, 2012.
   (A good introductory book on Machine Learning. Quick on some subjects and does not really cover data mining. But a very original and often quite illuminating perspective.)


•  Hastie, Tibshirani & Friedman : The elements of statistical learning. Data mining, inference and prediction. Springer, (2nd Ed. 2009).
   (A bit demanding, but very informative. The form is outstanding.)


•  Haykin S. : Neural netwoks. A comprehensive foundation. Prentice Hall, 1999.
   (An incredible source of information)


•  Haykin S. : Neural netwoks and Learning Machines. Prentice Hall, 2008.
   (Different from the 1999 edition, and yet still remarkable, and worth the reading.)
•  Murphy K. : Machine Learning. A probabilistic perspective. MIT Press, 2012.
    (The title says it all. A book incredibly comprehensive from a Bayesian perspective.)


•  Murphy K. : Probabilistic Machine Learning. A introduction. MIT Press, 2022.
    (The new edition, first of two volumes.)


•  Murphy K. : Probabilistic Machine Learning. Advanced topics. MIT Press, 2022.
    (The new edition, second of two volumes. Lots of recent works. A little bit of a patchwork.)


•  Sejnowski T. : The deep learning revolution. MIT Press, 2018.
    (Provides a personal historical perspective. By someone who should have shared the 2024 Nobel Prize.)


•  Zou Zhi-Hua. : Machine Learning. Springer, 2016.
    (A very good introductory book)

Books on specific topics:

•  Bishop C. : Neural networks for pattern recognition. Clarendon Press, 1995.
   (A good introductory book. Oriented towards connectionism but discusses many important issues in Machine Learning as well.)


•  Cristianini N. & Shawe-Taylor J. : Support Vectors Machines and other kernel-based learning methods. Cambridge University Press, 2000.


•  Webb A. : Statistical pattern recognition. Wiley, (3rd. Ed., 2011).
   (A good introductory book that deserves to be better known.)
  

---

Guidelines for the projects:

This year, your assignment in the projects is to choose an article from the list provided below, understand it, and, most importantly, try to replicate the experiments.

Rules are as follows:
• Projects are conducted by team of 4 students. Team members are responsible for decompose the work so that all members contribute meaningfully to the project.

The following deliverables are expected:
1. Before ( January XXX 2027 ), the choice of the paper and the names of the members of the team.
2. ( February, XXX 2027 ) Final report : 10 pages (strict. Reports longer than 10 pages will not be read!)

The intermediary report and the final report must be turned down in the ICML (Int. Conf. on Machine Learning) format for papers.

The deliverables will be evaluated taking into account:


• The clarity of the analysis of the chosen article. The paper should briefly present, in no more than two to four pages (out of the 10 allotted), the problem studied in the article, the main ideas and statements presented in the paper. The remaining pages will present the experiments carried out by the team, the possible difficulties, the results obtained and a comparison with the results presented by the authors of the article used as a basis for the project.


• The rigor and the extensive character of the analysis and/or the experiments carried out. A project that really answers the questions and possible doubts of the reviewers on the interest of the method and on the announced performances will obtain a higher score.


• Expression, clarity of explanation and quality of exposition. Reports can be written in French or English, as long as they are clear and well written.



---

List of the papers to chose from:



---

Projects chosen by the students:

Names :	Projects :
Student_1 Student_2 Student_3 Student_4	(Conference) Paper title. (final report: YES NO )




---

Propositions for internships :

 

sujet	Institution
sujet	Institution
sujet	Institution
sujet	Institution
sujet	Institution