Intensive Teaching · September 2026

The most important things

Before: Join the Iliad Discord and check you can see #intensive-september. That is where you post any material you need to share with participants. If you can't see it, tell Leonard (@leonardbereska on Discord).
During: At 17:40, make sure everyone fills out the feedback form.
After: Fill out the end-of-day log: rate the day and tell us anything that went wrong.
Fill out Who stood out: name anyone who made a positive impression.

Questions about pay, travel, logistics or materials: see the FAQ.

57 project proposals are open for review. Stars, a sentence, or both. Read them.

Your days

[D.7]World Models Fri 11 Sep, leading 39 responses · energy 3.7/5

Overall +0.30 better than expected n = 37

Content +0.05 as expected n = 37

Clarity 0.00 as expected n = 37

Motivation +0.05 as expected n = 37

Exercises +0.08 as expected n = 37

Readings +0.25 better than expected n = 12

Other activities +0.32 better than expected n = 19

Pace -0.19 about right n = 37

Difficulty -0.14 about right n = 37

Novelty +1.14 mostly new n = 37

Read what they wrote54

Worked well14

i enjoyed the group exercises
The last bit about cognition in cognition in LLMs. I hope we will have more focus on similar experiment driven research.
The fact that it built on top of yesterday's material was crucial to its value. But it was a very technical day that required more time to grasp.
The exercises let me understand exactly what was happening. The slides were gradually presenting information, making the lecture much easier to follow.
- The interdisciplinary angle i.e bringing in stat phys / neuroscience (a lot more intuitive for me_ - The Colab exercises!
Fernando did a great job at explaining very technical concepts and also in giving intuitions when faced with questions.
I think the slides for the lectures were well-designed.
I thought the introductory lecture was the best introductory lecture we have had so far. It motivated the framework in a clear way without assume too much background.
- The introductory lecture at the start of the day. Spending one hour to sent context before doing anything technical was great - Margot's talk at the end of the day is a great addition. Having someone talk about what they've worked on and the sort of work being directly applied to AI safety was nice.
Presenters were clear and concise
Mixing pen-and-paper exercises in steadily increasing difficulty with coding exercises.
It built upon the notions that we saw the day before!
The last part on metacognition seemed quite interesting (maybe not too mathematically though).
The real-world examples by Fernando, such as "Listening to a conversation from behind the wall" -- precisely what I asked for yesterday. They were helping putting the discussed material in context. Also the Colab exercise helped understand the topic and rebuild the math behind all this, very good idea. Also, the lecture by Margot was very satisfying as the day closer, where we went back to the high-level thinking.

Would improve26

for the group exercises we did some times we didn't spend time to discuss our solutions
Lectures were great. I suppose you can still feel whenever the lecture is taught by somebody with and without a substantial teaching experience. I'd like to spend a little bit more time to comprehend everything. As a meta-comment, it might be useful to split all the lectures in two days between two speakers. For example, the first part of the day is dedicated to the old material from the previous day, while the second part of the day covers the new lecture material. The idea is to allow people solve exercises a bit more, think of some questions and ask them the next day.
i'm not sure, but it would be cool to show/include more AI safety relevance; i felt that we basically just did the same math as yesterday in a slightly different setting and it felt a little bit pointless (<- that's a too strong word actually) to me?
There was quite a bit of overlap in concepts between yesterday and today.
As I suggested on Day 1, I firmly believe that a series of two-week "Iliad workshops" or schools on each of the topics covered in the Intensive would be extremely valuable as a whole.
Put the material online the day before. The exercises could be a little more wordy as I had to deduce a few things that could have just been written down.
- sharing the slides (incl. the handout version) and the Colab link before the lecture, instead of only via the QR code in the slides - the units felt a bit disconnected, a short recap tying them together at the end would help
It would be nice if the content and presentation of this day was more synced with the previous one. In many places, explicit comparisons could have been made as to how e.g. the formulas relate to those we saw yesterday, and I expect that this would have cleared up a lot of initial confusion and make the lecture easier to understand overall.
I enjoyed the content, the only change I would make is spending more time on modeern world models and why they work so well (e.g. what we know about "good" latent spaces for generalization and extrapolation etc), rather than the technicalities of epsilon-transducers - only given that the day was titled "world models".
I thought this was very similar to the module on computational mechanics, and it felt a bit repetitive without adding much insight. I thought the exercises were too easy.
There was some overlap with the previous lecture, I think this could be improved freeing up space for more material. The later part of the lecture was very interesting, as well as the introduction.
I wonder if this connects with other aspects of alignment, such as training dynamics or physics of deep learning. It connected with the topics of yesterday and was useful that they were built on top, but not sure what the advantages and things that the worldmodel approach miss are.
I would prefer if the teaching days were restructured. I personally find that having a single day on a topic is a bit too much of an intense burst. I would prefer if the day was cut into three parts, each with a different speaker. That way every speaker would be given the same time but it'd be spread out over 3 days. This would help with 1. Repetition: We would return to topics several times, helping with retention 2. Focus: Adding diversity to the day would help keep stuff interesting, especially towards the end of a long day.
Begin with a short summary of the previous day, and perhaps give more intuition for what we are trying to achieve
less today specific but about the entire week. we tend to have discussions every day about how the material of that day connects to ai safety and there seems to be a lot of confusion around it. like for example, there seems to be the somewhat implicit idea that finding the believe state in a transformer helps ai safety by understanding better how they represent data. but it would be very super cool to hear every lecturer explicitly explain why they think this is the most impactful way forward to make ai safer. currently it often seems a bit like research is blindly pursuing several directions that could maybe be relevant to ai safety. concretely I would wish for every lecturer to at the start motivate how this fits into ai safety and motivate their field from there, then present their field, and then at the end tie things back into ai safety and outline what they hope can be shown eventually and explain why this would help ai safety a lot. I wold also love for the lectures to explain why they have chosen to work on this particular field rather than other fields and how confident they are that this very field is the right one or if they're also somewhat unsure about which field could be impactful
I felt like we spent a lot of time today repeating the same material as yesterday, with slightly different notation and very different vocabulary. I didn't really see the point of introducing all this language for obvious-ish concepts until we got to the Factored Representations paper. So perhaps foreshadowing the relevance to LLMs and AI safety earlier in the day, and spending more time elaborating on this material, would've been better for me. Perhaps a few examples illustrating whichever definitions were necessary would've been good too. But I also had a headache all day, so possibly my outputs on this feedback form are not causal consequences of my perceptions or predictions. Please take this feedback with appropriate salt.
Just some notational inconsistencies. It was a great day.
Add corrections to the exercises in appendix
I think some diagrams were difficult to understand. I would have enjoyed discussing J-lens in more detail (also focusing on the actual maths behind J-lens) - I think the maths is kind of unprincipled and makes me question the validity of the claims made in Anthropic's paper. I would have loved discussing it at Iliad.
I thought it was a bit of a mumble jumble. We covered generic "world models", then went to computational-mechanic flavoured stuff and finally did a bit of a lit review on metacognition work. I don't think this worked out. The content has to be better designed and organised.
the lecture felt scattershot, unfocused and lacked rigorous definitions (which IMO are essential to understand the concepts covered today, by unpacking them and giving concrete examples). There were several errors in the slides that hindered understanding. The connection with yesterday's materials (computational mechanics by Xavier) was made but not as well as it should IMO. Also I don't see why world models were necessary for the lecture.
Last 2 presentations were too fast and I would have preferred a more explicit link between the theory and the applications (last part of the day)
Some more explanation on how can human contribute to these research topics competing with LLM. Why we still need human for these tasks, and in what sub-topic would human-participation would be especially valuable.
Slides numbering. Also, I think we should normalize the "Ohmmmm..." introduced by Julian so that we can finish breaks with it more smoothly. MSP was not defined before giving an exercise. Exercises were way too abstract in the beginning, but did improve significantly quite soon. The material itself resembled yesterday but felt much more abstract conceptually. The intro lecture had all this notorious RL formulae that are hard to interpret on the fly, especially without examples that map what formulae convey with each symbol. Part II with transducer for me was hard to follow because despite great examples given by Fernando throughout it felt very abstract and disconnected from reality still. It might well be that my brain is just not made for this kind of abstraction specifically though. Part III was much more computational and accessible to me, but some concepts (MSP) did not look too well defined to wrap my head around them. Finally, the produced fractals, as a result, did not click. Yesterday is a good comparison -- I was in awe. Today, however, I was utterly lost on what makes this promising.
nothing
Very minor thing: would be helpful to make the terminology between yesterday on computational mechanics and today consistent (belief states vs MSP; i don't thin they were explicitly called \epsilon-machines yesterday?) Otherwise, I thought today's lecture worked really well with yesterday's lecture.

Changed my mind11

world models are the next big thing!
For a long time I've seen work on agents as either theoretical that narrows to tabular RL (which is kinda boring) or purely empirical. The lecture and connections to computational mechanics as well as discussions on world models helped me reassess some of these views and think of new connections, especially to tensor computations I've been working on for a while.
I updated my definition of a world model.
The J-Lens!
I found the world model material very interesting and I plan to look into this material further.
n/a
it seems there is some real depth in this research agenda of computational mechanics. compared to mech interp I like the idea of building up synthetic data as ground truth and studying whether an LLM represents this data in the optimal way which it seems to do. I am still somewhat confused about a lot of the concepts covered today but it feels worthwhile digging deeper
I am more confident that this field has impact on AI-alignment and safety. While finding it fascinating, I was afraid it was too theoretical to be useful in the short term.
It motivated me to check some frontier papers. Also discovered some AI programs throughsome other participants.
This side of comp mech did not convince me to be worth pursuing, most likely because, as Fernando mentioned, it's hard for the bottom (Transducers) and the top (Metacognition / personas) to meet in the middle, at least yet.
I believe this question should be written 'Did you change your mind about anything today?' or 'How did your mind change with regards to a certain topic' or something. Anyway no idea still figuring it out

Other thoughts3

n/a
Today's schedule is a bit tight
To make it more general: I think we should normalize the "Ohmmmm..." introduced by Julian so that we can finish breaks with it more smoothly. As otherwise lecturers need to be more loud to call people in, and that is not as effective

[D.1]Decision Theory → RL Mon 14 Sep, leading 38 responses · energy 3.6/5

Overall +0.14 as expected n = 36

Content +0.28 better than expected n = 36

Clarity +0.08 as expected n = 36

Motivation +0.14 as expected n = 36

Exercises -0.25 below expected n = 36

Readings 0.00 as expected n = 13

Other activities -0.11 as expected n = 9

Pace -0.17 about right n = 36

Difficulty -0.22 about right n = 36

Novelty +0.67 mostly new n = 36

Read what they wrote50

Worked well17

- Grounding RL in preferences instead of just starting from rewards
The exercises with solutions. The formal introduction of the axioms with examples.
Less coding exercises: I think that this was a good choice given the material.
I liked the structure of today's lectures. Starting from the fewest assumptions possible and building up to something that has enough structure to prove results is very intuitive.
Exercises like Problem 2 where we actually did compute some stuff, plus a few other like Problem 1 which taught us how to derive the Bellmann equation -- helps invaluably.
I really appreciated the first lecture. I have read intro books on RL before but never delved into how the structure of preferences determines the reward function. Very interesting!
exercises! very fun to do some math
I really enjoyed the exercises.
We had all the materials sent beforehand
Lots of exercise time, Leo was very helpful in particular.
vibe-reading by my own the materials
I think the slides were structured well and there was a natural logical progression in the results being taught.
i liked the agency part surprisingly much
The logical structure of how stuff was presented made sense, we built well from one idea to another.
The initial discussion about what agency means, and how we should model it, and the introduction of the von Neumann framework.
not sure, spent the whole day being pretty confused about preference and utility
I thought the the connection between the fundamentals to the broad strokes of RL was illustrated with a lot of clarity.

Would improve22

I felt like introducing such practically relevant content in terms of axioms was a bit terse. I would've enjoyed it more if a more practical approach was taken to the content
More concrete examples
Good class today, but it felt too self driven. The lectures were relatively short (2.5 hrs) in total maybe, and the rest of the time was dedicated to independent exercises. I think it would have been more useful to maybe cover some relevant problems or applications as part of the lecture.
- Would have liked a paper or textbook chapter to go with the lecture (e.g. Sutton & Barto ch. 3 for the MDP part) to have something more formal to work from - More worked examples connecting the VNM axioms to reward / return / policy
The day was quite abstract and very proof-based rather than compute-based. I personally was quickly losing interest in the exercises as I did not see value in doing those proofs. Perhaps would be worth considering to have problem sets for differently inclined people: smth like Problem 2 but longer and much harder for applied mathematicians / theoretical physicists, some Colab notebooks with simulations / visualizations for more computationally-inclined students, and indeed proofs for pure mathematicians. Finally, just a selection of conceptual readings like LW posts or arxiv preprints to see what are those non-Euclidean preference axioms in practice for more philosophically inclined ones. Also, what I lacked was a more empowering example. Maybe it is hard or even intractable to provide one, but say showing how it works in say AlphaGo or another familiar system would be helpful to see the value in this. Also, the notation in problem sets: what is that infinity under the || metric?
The day was too sparse, too much of unstructured activities
This lecture was much better than last Friday's. It covered concrete set ups and I felt like I had some sort of closure after each of them. However, this feels quite disconnected from how people do RL, ie. it would've been useful to do some coding (we have covered the Arena material essentially so maybe presenting the corresponding coding exercises as alternatives to the theoretical exercises would be useful). Also, I am not too sure how RL is done in practice, ie. how does deep-RL work? How do we optimise policies in these cases, etc.
Add a third lecture - more material!
Liked the introduction to rewards in the first part, I was wondering if it can be compared to other approaches and discuss a bit more where it fails/what it can't capture.
maybe some discussion about the exercises as a group so we keep the energy more focused, but the exercises + lectures mix is personally so much more effective than lectures alone
I particularly enjoyed the second part. I kind of see the motivation behind the first part, but felt the assumptions where quite unrealistic, the notes were great, though.
I would have liked a few examples of contemporary research questions around the topics we covered today
I especially liked the first part related to introduction to utility functions and derivation of the common reward optimization framework from the common assumptions. This part was definitely valuable. Overall, I tend to believe that the core part of learning and teaching given that we all have LLMs right now is an ability to identify corner cases, limitations and point people to them. On the other hand, it might be useful to dedicate at least some time to Deep RL and some ideas and limitations that break in when we start dealing with finite datasets, off-policy learning and functional approximation instead of purely tabular settings.
Too much time for the exercises. I would have liked more explanation and discussion time.
I think maybe a few more slides on Q-learning, as it was a little more difficult but seems like quite an important aspect of this theory. I thought the exercises where quite easy apart from Ex. 6 which was quite tough.
i would have liked solutions for the exercises or discussion for what we should take away or should have taken away (afterwards)
I liked having the schedule for the day on previous days. One could also consider splitting up the lecture time more, since the exercises ran very closely along the lecture, so alternating more frequently good be helpful, as it would ensure that previously established concepts have been internalised before moving on. As it was, the wait between "you will do this in detail in the exercises" and the actual exercise was a bit long for later topics.
I found today's material pretty interesting; I've seen a lot of it before in some form, but this was a clearer and more complete presentation. However, there might be some room for improvement in pedagogicalness. To some extent it suffered from the classic "math's pedagogical curse": Equations and proofs were clear, but it took some work to read the underlying concepts/motivations out of the equations. For example, maybe some schematic discussion of MCTS would've helped give a clearer intuition for the Bellman equation? And certainly vNM/Dutch Books lend themselves to cute examples about drunkards and gamblers with irrational preferences. Generally any excuse to put a diagram on a slide full of equations sounds good to me.
- Maybe swap one of the analytical exercise sessions for a hands-on implementation of RL in a simple setting (e.g. where the set of states is small) - I would have liked to see more in detail when RL in the von Neumann framework fails, and what are the open problems
The language and notation in the exercises are difficult to understant. It's a pretty mathematical day as well, I'm not sure I saw the direct link to AI safety.
I would try to have more practical examples for the material. Plots trial systems, anything more practical to illustrate the math.
I was already familiar with most of the material and so I decided to read some papers from Simplex during the lectures/exercises.

Changed my mind9

I appreciated in particular the pre-lecture reading on decoupling beliefs/desires. That persuaded me that our theory of agents loses little generality for the common-sense meaning of agency.
That the entire field might be as misguided as Economics due to expected utility theory. The ASI most likely won't have consistent preferences because you don't need them, just like humans survive easily without them. No one Dutch books anyone or money pumps anyone because the effort to discover the inconsistent loop requires NP-hard computations, which no one ends up doing. Thus undermining the motivation behind the field and most justifications.
n/a
really curious now to learn how Q-learning was applied for the bot that plays starcraft
I discovered that even though RL is not the main technique frontier models rely on, it might be helpful to understand it because it is used in post training.
agency is much weirder than I realized, i find the vNM axioms much less obvious than before
RL really doesn't tell us much about alignment per se!
Did not get time to reflect on anything
RL can be worse than supervised learning in a lot of cases.

Other thoughts2

The day

10:00 Lecture starts

12:30 Lunch at LISA

13:30 Afternoon session

17:40 Feedback form

18:00 Day ends. Dinner at LISA

When	Where
Mon 7 Sep	LISA, 25 Holywell Row, London EC2A 4XE
Tue 8 – Mon 14 Sep	23 Curtain Road, London EC2A 3LT
From Tue 15 Sep	69 Wilson Street, Moorgate, London EC2A 2BB

Resources

Elsewhere