Intensive Teaching · November 2026

The most important things

Before: Join the Iliad Discord and check you can see #intensive-november. That is where you post any material you need to share with participants. If you can't see it, tell Leonard (@leonardbereska on Discord).
During: At 17:40, make sure everyone fills out the feedback form.
After: Fill out the end-of-day log: rate the day and tell us anything that went wrong.
Fill out Who stood out: name anyone who made a positive impression.

Questions about pay, travel, logistics or materials: see the FAQ.

Your days

[D.1]Reinforcement Learning and Misgeneralization Wed 4 Nov, leading 55 responses · energy 4.1/5

Overall +0.69 better than expected n = 54

Content +0.94 better than expected n = 54

Clarity +0.52 better than expected n = 54

Motivation +0.56 better than expected n = 54

Exercises +0.11 as expected n = 54

Readings +0.57 better than expected n = 21

Other activities +0.28 better than expected n = 18

Pace -0.20 about right n = 54

Difficulty +0.11 about right n = 54

Novelty +0.91 mostly new n = 54

Read what they wrote65

Worked well18

The lectures were really engaging, and included useful background and comments which helped me orient myself and get a feel for where everything sits. There were lots and lots of questions slowed down the lecture, but this worked perfectly. It was just a function of everyone being so interested and wanting to understand fully the material. Also, it was fun to sit around together and work on the problem sheets during the breaks. Just really productive in every way. I wrote the material was 'mostly new' to me above, but in some sense it was all new, just that, because Fernando posted the slides on Discord the day before, I had a good look at the first lecture yesterday evening.
The topic was about theoretical framework for treating the various aspects about agents (embedded agents, non-embedded, etc.) Many things that we discussed where topics I have given very little thought previously -- much less than they deserve! The session helped me to appreciate *many* questions related various configurations of agents that I was overlooking previously. Ashe is an excellent communicator, and the way, and subtlety, with which he was approaching our questions is something I can learn from.
Very good lecture notes and slides and exercises. Especially the notes are useful and enjoyable to read. I think notes with good condensed information are the best way to learn
I liked the mathematical detail provided by the lectures. In particular the fact that more complexity was introduced gradually, with clear explanation of the motivations for doing that.
The exercises
I just think it was a really, really interesting topic. This is more what I came here for. The first two days were much more "soft" topics which I think most of us have already been exposed to at this point. I was really really happy to learn a completely new, relevant topic in a way that I wouldn't necessarily be able to learn myself. I was also really grateful to get more clarification on what RL really is. It clarified a lot of the discussions and debates that I have heard among machine learning experts.
The lecture was absolute cinema. I understood everything and now I am very motivated about RL. Very well explained the content. Also the open question in the worksheet were interesting.
The teacher is great. He knows how to organise materials in a logical sense and how to present them to give as much as intuition as possible.
Very good content!
Excellent lecturer! I feel like this was the absolute perfect level for teaching mathematicians with no RL experience.
The lecturer was friendly and welcomed all questions
Instructor really emphasized big picture vs details and told us where this was opinionated vs standard. Very helpful for organizing knowledge.
Everything had very well defined mathematical background. So had a concrete goal.
The instructor answered questions very well, and rather patiently.
The material was on a topic about which I had solid intuition and background knowledge. That helped to participate actively by asking clarifying questions and participating in discussions. As a consequence, I understood aspects that previously were not clear to me and which, I believe, will stick to memory for a long period of time.
Good structure! Good division between lecture time and time for exercises.
As I mentioned, plenty of audience questions meant a lot of the ambiguity was clarified, even before I had the chance to realise.
The second set of exercises was amazing.

Would improve33

It was amazing, I learned so much!
I enjoyed the session very much. One thing I would like to see different is more worked out examples that capture the subtle aspect of the abstract concepts and help to understand the motivation behind introducing them better.
Too many questions are being asked, spending a lot of time on peoples personal ideas/thoughts/issues and interupting the flow of the lecture. Especially when people ask questions which will be answered in 2 mins in the lecture anyway. I think one should limit questions during lectures and postpone them until after to direct discussion with the speaker I would've also liked to understand how RL is actually done in practice. I feel like we've seen the theoretical background but I doubt that actual post training for LLM's fits in this and I would like to know how it is done there.
I wish there was a separate day where we could just work through some of the exercises, either individually or in small groups, with the aid of the lecturer when needed. It would also be great to see how to apply the theory to some concrete examples or open problems.
The only thing I would improve here would be to have time to look over the material before the lecture. It's pretty technical and there is a lot of it.
Would have been nice to have some programming exercises.
Today we had lectures that lasted several hours without any breaks (most studies show that people can only meaningfully focus around 1 hour). It made it very hard to follow
I loved the content, finally bringing the rather empirical work we read up to today with some robust formalism on RL. The way the session was conducted, however, could benefit from a couple improvements. Specifically, the two things I wanted to point out were (1) not having consistent breaks (as we did on Tuesday) was very tiresome, especially given how heavy the material was and (2) I feel like the speaker was a bit too permissive on the amount of questions asked, which prolonged the session by probably almost 2h overall (throughout the whole day). Having discussions is a nice thing, but in excess, it felt like it discontinued the overall flow.
Maybe I'd like more examples (posibly computational) or some exercises in the worksheet involving some interesting particular case.
Very little, the lecture was great.
It's good that there is plenty of engagement with the speaker and there are often good questions, but some participants are interrupting the speaker with questions and inappropriate times, or making unnecessary comments. This slows down the speaker, and I find it interrupts the learning for me. Could an announcement be made to at least put up your hand and wait for the speaker to be ready?
My only criticism is that I think that we spent a little too much time stuck on answering participant questions, so the lectures ended up running more than twice as long as expected. Some of the questions are more comments or observations and could probably be gently encouraged to be kept to a minimum.
In the opening of the session I'd give an overview of 1) what is going to be taught, what the new concepts are, 2) what the prerequisites are and 3) why it matters.
I think that starting with the formal beliefs/preference framework before RL is not the most pedagogical. Introducing RL first could motivate a more axiomatic approach later.
Lunch
May be a bit motivation and quick recap of a dictionary of basic stats to Linear algebra. But not much everything else was fine.
It was excellent. I would just include some short break in between
The lecturer and the lecture material were excellent. However, one particular participant, Arman Darbinyan, interrupted every single slide with 3-5 overly pedantic questions, frequently interrupting without even raising his hand. Overall his questions were about minor/irrelevant notation issues which disrupted the flow and was very counterproductive. Each lecture ran overtime by 1 hour because of this. I suggest to leave questions to the end of the presentation, or to maybe allow for questions every X slides, otherwise the rest of the program will be impacted by this behaviour. Regarding exercises, most of them were focused on deriving or proving certain mathematical properties, which might seems targeted to mathematicians. I am an engineer, so I would have benefitted more from a practical exercise, like implementing value or policy iteration for a toy example.
The lectures were pretty good for the most part, but they were delivered in 1.5-2hr blocks which made it hard to assimilate.
The morning lecture formalization seemed a bit dubious to me, coming from physics. I would have preferred a more intuitive explanation on what was really going on. But the afternoon part was very clear and I liked it.
It would have helped to have one or two concrete examples while going through the material
Overall, I think the class today was very good.
lectures were too long and dense; consider having breaks in between, like after 30 minutes; perhaps incorporate discussions into the teaching. otherwise it's very hard to sit there sitting still for 90 minutes concentrating on a single lecture. in terms of content, i feel the mathematical model of RL could have been better motivated; also, fewer topics but deeper
The session was very informative and helped me to forge my knowledge and intuition about the topic. Most importantly, it helped me to connect the knowledge that I had from math to the AI safety context. What I would like to see different: There was a problem session, which was a very fun part, helping to play with the concepts and understand them deeper. However, some of the definitions for the problems were different from what we had during the lecture (i.e. which were on the slides), which led to unnecessary confusion. Some of the slides contained not consistent definitions and math notations as well. Personally, I found the presence of those flaws quite helpful too, as it allowed me to reflect deeper and challenge my understanding before accepting them. However, I believe not every audience could appreciate it. Another thing that I would like to see being done: a brief discussion of the solutions of the questions or providing the solutions as a hand out after the session. Overall, it was a very enjoyable and helpful session!
Nothing! Great session!
The lectures today were great and generally really well presented (I think this sentiment was echoed by the extensive questions from the audience). One improvement is that especially with the second half, perhaps there was a natural breaking point (nothing long, maybe just ~5mins) to fully digest some of the ideas before going onto a separate topic.
There were a few points that were unclear: for instance, pi moved from being a probabilistic policy to a deterministic policy. Some interesting technical points were omitted, e.g. turning pi from a probabilistic to a deterministic policy was suppressed. Ime the absence of such things is typically more confusing, so it may have been better to just speak of deterministic policies. When the ordering of lotteries was introduced, it was initially unclear whether an order was being introduced with certain properties, we were outlining properties that such an order should have, or otherwise. I think the slideshow would've benefitted from far more signposting, making preamble clear e.g. "consider an order, we'll talk about desirable quantities. let mu be a measure such that" and so on. I would have overall preferred more examples. The speakers answers were comprehensive, however, but some of the questions should have been addressed by the slides. Quite a lot of time was dedicated to the exercises, but they were not discussed. I think they were worth doing however It felt like sometimes we got in theoretical rabbit holes and the big picture was a bit blurry, but that may just be me
More of a comment on the interaction with the audience than anything else, but there was a lot of time spent in tangential discussion that was pushed by audience members I thought could have been better spent! Some single slides were discussed for about ~30 minutes.
I thought the lecturer was fantastic and the material was great. I did however think that the lectures went on too long. This meant that we had much less time to work on the exercises, which I think would have been the best way for us to learn, and it is also difficult to maintain concentration for 2 interrupted hours. I think the cause of these long lectures is not the fault of the lecturer but just because there are probably far more questions that were anticipated. Perhaps this is due to the cohort size and lecturers using the same notes as for previous, smaller cohorts? This was also a problem yesterday. It would be great if then maybe taking questions was done slightly differently and/or there were more breaks and/or their is less material covered.
I imagine most of the audience is familiar with vNM-type frameworks. Maybe better to spend less time on this, and more on the less familiar stuff in the second half.
I thought that the plan of the lecturer was excellent. However, I think that allowing unlimited questions and interruptions slightly derailed the plan. I felt that I would have gained more, if the allocated time for lecture and exercises had been enforced more strictly. This is more an issue with some of the students deciding to ask many questions during the lecture time rather than afterwards, but I still think that the lecturer could enforce a little more rigidity and perhaps after a certain amount of time is lost, to ask students to leave all remaining questions until after the lecture is over.
The discussions during the lectures were a bit too long. It is good to clarify a few things at the moment, but a 10 min discussion at the moment we have not yet defined almost any structure... one looses the track.
Would be nice to have more circular areas / sofas for discussion instead of the array of desks. I don't think external.monitors are necessary

Changed my mind9

It helped me to appreciate much deeper the role of abstract models in providing frameworks for studying agentic foundations and look at possible challenges from a unified angle. It opened up many directions in which I can study the topic deeper and motivated me to spend my upcoming weekend to read more on the subject. It also helped me to think more about how to define world models as a mathematician, and how to consider the agent related questions in a theoretical frameworks. This is something I want to continue doing further.
About utility functions. The axiomatic approach really let's you understand what goes behind utility functions. Fernando orally suggested reading "About the independence axiom" which I found very interesting.
Mostly I just understood more what RL is about and what I understood a little better what the different perspectives where.
Agents can be defined (somewhat) mathematically, and now I think I can think on philosophical questions about them.
There's interesting problems in RL for mathematicians to work on!
I thought RL was simply a recipe that you can implement and post-train a model in a few lines of code. It turns out it's a "bit" more than this!
This was perhaps mathematically the most fun it has been so far.
I thought deeper on the underlying learning mechanisms of RL models and on how rewards and utilities can be modeled, which helped me to think more about boundaries of safe and reliable applicability of the current RL approaches. Most importantly, I feel more motivated to dig dipper and learn more!!
Reinforcement learning seems to have a somewhat tractable way to talk about objects that might be relevant to alignment (referring explicitly to model “preferences”)

Other thoughts5

I want to thank Ashe for his time and effort. It was a very enjoyable session, and I learned more not only about agentic foundations but also about how to communicate more effectively in similar settings.
no
After the session, I feel more motivated to dig dipper and learn more!!
N/a
I would really appreciate it if there was a 'moderator' that would notice if the lecture becomes derailed, and would intervene on behalf of the lecturer if that happens. I think that a significant part of whether the day is very beneficial or not, arises from whether or not the lecturer's original plan and time allocation comes to fruition. Also in day 1 (though for different reasons) there was insufficient time for the exercise portion, and that makes a big difference on the learning process.

The day

10:00 Lecture starts

12:30 Lunch

13:30 Afternoon session

17:40 Feedback form

18:00 Day ends

When	Where
Mon 2 – Fri 6 Nov	An office near LISA, address to follow
From Mon 9 Nov	LISA, 25 Holywell Row, London EC2A 4XE

Resources

Elsewhere