Every cohort

Student feedback

Abstractions and Latents 3 runs 60 responses, 86 written

for pasting into Claude

September 2026 Mon 21 Sep Satya Benson 37 responses energy 3.5/5

Rated against what each person expected. 0 is as expected.

Overall -0.94 below expected

Content -0.81 below expected

Clarity -0.58 below expected

Motivation -1.03 below expected

Exercises -0.30 below expected

Readings -0.67 below expected

Other activities -0.54 below expected

Pace -0.29 too slow

Difficulty -0.16 about right

Novelty +1.13 mostly new

Read what they wrote47

Worked well9

Reading through the notes by myself.
having the time to read and do the exercises at my own pace
Lecturer communicated clearly.
- Reading a section and then doing the exercises straight after - The small group discussion
Speaker was clear.
Aron (?) seemed far better at explaining and communicating.
I think the concepts are very interesting and probably something I'd like to think about more in a couple of days when proposing a project!
Formal distinction between mediation and redundancy.
The structure: lecture->exercise->lunch->lecture->exercise->discussion

Would improve25

Improve the teaching and maybe the material selection
Fewer definitions and theorem proofs, more application of methods and experiment results.
To me, the connection to Alignment was a bit haphazard, contained in only a few bulletpoints at the end of the notes. I would prefer it to be more center stage, perhaps a presentation + discussion, could be very short. As it was, it felt very vague and swept aside. I found it difficult to focus during the lectures, partially because they were slow, but didn't go much beyond the formulas. I think they would be better if more intuition for what things mean (including questions like "what does this entropy tell us?") was explained.
There was no motivation at all on why this matters. I did not like the teaching style. Exercises in the first half just for working with notation and really easy.
The slides seemed completely AI generated and the concepts were not very well motivated and I think there was a shared feeling among participants that we didn't learn anything new.
Since some point of the intensive, I feel like we are manipulating the same concepts, but using different notations. I feel that there is a big overlap among courses but these frictions make it confusing and the short time that we have to familiarize with each topic everyday make it so that we cannot really delve deeper in the different theories.
There was very little content today. Most of the content that was present seemed to be information theory and very few new ideas and connections to AI Safety. It would be better if more new content was included. If there isn't much to be discussed as this is a very new field, it should be scrapped/merged with another similar session.
1. H needs to be defined with a name on the slide for entropy 2. questions were helpful in the lecture to understand the setup better, 3. in the notes label the conditional mutual information 4. add more motivation and ai safety context, how can these be tested/computed
I think this literature (which is very niche) could have been positioned better against the wider literature on latent variable models and latent identifiability in ML.
So many things. This day makes no sense. The research direction is very flimsy, the delivery was not very good and it's just difficult how this fits the alignment and safety direction. Iliad needs to seriously reconsider its curriculum to cover more in-depth established research programs like SLT.
Although the lecturer presented clearly, I think the material was not well motivated. The basic problem we were trying to solve - how two agents can agree on a set of 'beliefs' - should have been more clearly communicated from the very beginning.
- A concrete example of a natural latent early on, before the mediation / redundancy conditions - Pointing to the formal versions during the lecture (e.g. the natural latents paper) for going deeper
I'd like to see a slightly deeper theory + connections to sparse graphical models & linear algebraic algorithms to compute the things the theory cares about. At least for the simple gaussian models to get better intuitions.
The speaker was clear, but overall I think the material was not well motivated today. We should have begun with a clear description of what we were trying to achieve - i.e. how we could model different agents coming to the same set of 'beliefs' - and properly defined how the associated objects (observations, latents) relate to the problem.
I found the material very poorly motivated. The lecture exposition would have benefited greatly from a roadmap. The definition theorem format gave little insight into the 'bigger picture'
This material should be removed from the curriculum. It is absolutely not clear and not motivated why a less wrong blogpost from the last year should be taught for a whole day. Also the teaching needs to be reconsidered.
A bit more of a link to empirical things would have helped cement the information-theoretic foundations into a more practical context.
I didn't quite understand why we were learning this and felt a bit blindsided. I enjoyed the work on a technical level but still have little idea why it was useful for AI safety
I would like to hear about how alignment is done. We're halfway through the intensive and I feel like I've not heard anything about alignment
I think maybe a little more content on the algebra of information theory - the algebraic manipulations are not always very easy and even though I have some prior experience, I was defeated by the two-witness lemma.
The material felt very (nomen omen) abstract; I struggled to see how it could be applied to any more practical settings. There was also not much substance in the material, it felt like giving new names to existing concepts. Also, I think the lecturer was quite jet lagged and it negatively affected the flow of the lecture.
The talks seemed completely unprepared. There was little emphasis and clarity on why this matters. What is the big picture?? How do people use this with deep learning models? And for alignment?
Everything. This felt like an opaque subject that was not properly motivated in the beginning of the lecture. The exercises in the first part were ok to disentangle the difficult language but exercises in the second part were just "proofs left for the reader" which were not very useful to me. It would have made more sense to study real models where latents are concrete and useful. The lectures should include at least one example showing the different tree vs non-tree examples of latents like was presented in the notes. The notes themselves are better than Eisenstat's paper, but still need to be clarified with notation and language that is coherent with the rest of the Iliad curriculum. Working out an example on the white board would slow the pace down and help everyone understand. I did not follow the second part of the lecture at all. No idea what was happening.
More visual diagrams for the condensation framework.
This was too abstract, more examples needed. Not necessarily real life examples (although that is always welcome), but examples to each statement / theorem / computation would be highly beneficial during the lecture. Also, in the very beginning of the lecture we were just submerged into the subject unlubed. Having slides allows one to show the applicability or the empowering example or an impressive result or some gradual real-world examples that help us pick up the thinking so that we can translate it into formulae later. The intro does not need to be long (5-10 mins is completely enough), and I am aware an intro runs a risk of questions arising that drag the intro out into the philosophy (as the Day 1 of SLT has shown). But it has to be there to make the immersion gradual -- some people might have well woken up 15 mins before the lecture, while others, despite traveling there for 1 hour and being for 2 more hours on their feet, do need a gradual context switch. Also, the number of links that was mentioned for exercises was way too big. Like another Iliad website.

Changed my mind7

n/a
n/a
I thought the concept was very interesting and I can almost see how it ties into other topics. I would have appreciated a more holistic view on things though.
I feel a bit frustrated and want to know more about the purpose and direction of research, and would rather engage with that kind of material. These questions are fun but it feels like a poor use of time. It's hard to choose a research direction when you don't see why you're doing it. I feel this sentiment is shared, If the situation is so urgent, we should be engaging more with actual research and what is the state of the field.
Understanding condensation does not seem like something useful because we have no proof that LLMs understand concepts that same was humans do. Isn't that the whole point that superintelligent AI could be an existential risk? An example is neuralese!
Natural abstractions can be mathematically formalized.
That I probably have no wish to do dry abstract math

Other thoughts6

Have chats in person with some of us for ideas to improve the intensive. I genuinely think that its a great initiative but there has to be some structural changes to the program.
n/a
n/a
I was sick so I only attended the first half of the day
Solid foundational theory.
I will leave after lunch tomorrow because I have an onsite interview. That's why I can't make it to the London tour, sorry -- thank you for organizing btw! Must be very nice for non-locals.

June 2026 Wed 24 Jun Satya Benson 13 responses energy 3.0/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.1 of 5

Teaching and format 2.8 of 5

Difficulty match 2.9 of 5

Energy 3.0 of 5

Read what they wrote14

In their own words11

The first talk felt well-paced, the second talk felt a bit rushed, but the concepts cleared up for me with the successive reading session. The exercises had a good level of difficulty and were insightful to understand the topic. Mandatory homework reading exercises to prepare for the next day after the full intensive day seem a bit much though.
I don't think the speakers did a great job in exposing the contets. I found the motivations promising, even though hard to grasp as the speakers couldn't convey them in a clear way. The formalisation of the main concepts were poorly presented and they didn't convince me at all. The exposition lacked mathematical formalism, and many claims were either unprecise or unclear. I also think that some of the concepts presented today are already available in different areas of mathematical statistics–such as information theory and information geometry–from which they could have inherited definitions, notations, etc. Rephrasing the concepts in a standardised framework could help both students (who more likely are familiar with the standard frameworks) and speakers (who don't have to redefine the theory from scratch). Moreover, I believe that this could also help the field build upon settled knowledge, thus facilitating possible advancements. I would have appreciated if, when asked, the speaker had explained possible connections with the notion of the Fisher Information Matrix.
The slides were pretty bad, too much text and too small (especially for the condensation part, also speaking too fast for that part). The teaching and exercises were great! I like the website format much better, much easier to navigate than a google doc.
Pretty bad lectures. Daniel was able to explain things properly later. It is very important to get the mathematical structure right first, and then it is somewhat conceivable to do some handwaving expecting people to be able to fill those gaps, but at the very start, you need to be very concrete.
I have comments for today, from the teaching format to the content being taught: Today stood out as an exceptionally poorly motivated day, even though the general ideas clearly latch onto several lines of research in AI alignment. It was clear that the initial lecture meant to present a rough sketch, which is fine enough. We were presented, however, with some mathematical window dressing, accompanied by technical terms which were in conflict with the story being told. This naturally raises some confusion, which itself was met with downplaying the problems which were obvious to audience. I think this is a problematic way to introduce people to a new subject. Secondly, it's not obvious to me why one would like to spend an entire day on this framework itself. It seems clear that the main points are conceptual and can be quickly taught and elaborated on, whilst the mathematics is mostly vacuous as it stands. This was apparently already obvious many years ago. The theory leaves out many important considerations: How are the spaces for the representations being chosen and constrained? How and why would agents agree on observations? And I'm sure many others I'm not thinking of. And as a postscript: I'm quite stunned by the terminology used in the work. For instance, the word mediator already has a well defined meaning in statistics and causal inference. It's highly suspicious that this word gets hijacked in way that is incompatible with that widely used meaning. These are such adjacent fields, which we should hopefully have some knowledge of.
I felt like the introduced concepts were too abstract for me. I think more examples would definitely help with understanding of these ideas.
I wasn't able to attend the first half of the day but in the second half we did "Condensation" and finished exercises. The lecture in the second half was okay but not as engaging as other days have been. Abstractions and Latents seems like a fascinating topic and I'm intrigued as to how we'll build on it tomorrow.
The first lecture would have been more useful to me if more time was spent to make the relevant mathematical objects clear. I think the general motivation would have been better placed in the reading materials. I thought that the second lecture had too much text on the slides for them to be useful for the audience. I didn't do so many of the problems, but they seemed nice. The condensation paper is too long to read on your own given the time, so a rigorous summary should be made if it's essential content. I did see the blog post summary but found it unsatisfying.
- The new Iliad website is nice - The exercises were useful to clarify definitions and work through the concepts - It was nice to have a clear connection with alignment - Discussing in some more depth the different ways in which natural latents/condensation may contribute to safety seems appropriate; we are here to learn about possible contributions to AI safety after all - the morning slides were missing pictures and examples to help us understand the concepts - The example that Leon asked for (two strings of 100 coin flips; the latent is the bias of the coin) was very helpful - A reminder of the definitions of conditional entropy and total correlation; or the requirement for us to review the concepts before today's session would have been useful. I was not familiar with information theory at all and I imagine there were other participants in the same situation. - The concept of abstractions in the morning lecture was unclear to me - The concept of retrieval costs in the afternoon lecture was unclear to me - The afternoon lecture was generally too fast
well presented material, the game in the morning worked quite well nice discussions. Time management could be improved: started late, shortened lunch break and then random 25 minutes break, unclear schedule. Also, I think calculating expectation valued of Gaussians on a whiteboard is not a very efficient use of time, while more discussion time could be useful. Going through a couple of round of the game to see if teams can figure out their strategies given more input would be fun. But it was a really good day overall!
I don't think this is yet a well-formed theory. I don't think the set-up, the scope, the objects of interest etc have been clearly defined. I am not sure if there is some content in all this that justifies the time investment necessary to acquire this non-standard language. All I was shown are some basic manipulations of the definitions, of the type that follow any mathematical set-up. This is the kind of thing that would be great to discuss when introducing the subject. I also think that ignoring existing statistics and information theory does come with risks. I am not sure that classic work does not cover much of these grounds already, possibly from a slightly different perspective. I am far from an expert, but I see striking similarities between these concepts and main-stream ones. Putting this theory in dialogue with existing ones could allow one, for instance, to import well-known results and guard against known pitfalls. Ideally, this would be explored in a lit review but could not find one in any of the pre-prints and posts. The exposition was poor and confusing and the day unstructured.

Other thoughts3

I really do like the course so far.
maps straight onto my week's work, redundancy predicts cross-model sae feature transfer, mediation leg is the harder one. more on mediation estimators would help.
It would be great to have a copy of the program with the full names of the speakers. It's hard to keep track, especially when people are not introduced and do not introduce themselves.

April 2026 Wed 22 Apr Sung En (Daniel) Chiang 10 responses energy 3.5/5

Called “Abstractions” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.0 of 5

Teaching and format 3.2 of 5

Difficulty match 3.3 of 5

Energy 3.5 of 5

Read what they wrote25

Worked well9

The readings were interesting.
The exercises were informative, and I like the group discussions where read different topics
Good presentation at the beginning
Compared to C3 the amount of material (being less) matched better what felt like could fit into a day. I liked the format and could extract more value from the cross-pollination conversations yet the day still seemed intense.
Initial presentation
Good to get knowledge on some ideas in the field like natural latents
It was sunny. Now that we’re 2.5 weeks in, people know each other better, so they seem less inhibited when discussing, as if they have ample evidence to trust that mistakes and disagreements will be handled graciously within this cohort.
Good initial lecture, good readings
Exercise sheets today were probably the most relevant and informative of the whole of the intensive so far I got the feeling I was learning something very significant in the way I do during an actual maths class. Quite motivated to learn more or contribute to natural abstractions now.

What they took away8

That natural latency stays stable between agents with different internal models.
I enjoyed the section in [participant]'s writing about people attempting to NN in QFTs
The concepts of natural latent and condensation
Even with its limitations, the latents framework was interesting to explore. I have encountered it earlier and mostly dismissed it for being too idealistic and far-fetched, and after today i find it significantly more motivated and i can articulate my concerns with it better.
abstractions has a long way to go and is missing empirical demonstrations or studies of falsifiability
Concept of mediators and refunds was useful
I had a nice chat with Claude about the history of the term entropy (and its relationships with KL divergence and mutual information) during the exercise portion of the day.
Mathematical formalism for abstraction - and great group discussions on how we might test this formalism in “toy model” AI systems

Would improve8

I would add the solutions to the exercises so it is easier to check if I am going in the right direction.
Put more real-world motivation or draw more links with concrete things that have been done. I found it was harder to motivate myself compared to other days (although this could also be due to my energy level today and/or random motivation fluctuations).
My experience with the exercise activity was mixed. On the one hand not having the solutions, nor much of a summarizing discussion (well, we had some of that but really not much) it felt rather uninformative, similar to some early days (IIRC the deep learning days), with what seemed like math practice exercises and little to take away. It may have had to do with my energy somewhat decreasing towards the end of the day. But on the other hand, this prompted more small-group discussions than earlier, which was an enjoyable experience. Lately the TA involvement in breakout discussions has been fading. This makes me feel insecure sometimes about the direction the conversations go (regardless if i or others are steering it). It would be useful to have more of that!
extract falsifiable predictions from john et al. using enhanced interrogation techniques, dial back the scope
This area really seems to lack some experimental demonstrations, even for toy models. And I think people were less engaged in the material because of this. The questions of interest need to be constrained by actual examples and it doesn’t seem like this work has been done.
When Daniel lectures, he talks very fast. I wish he would speak more slowly. I would have liked it if he summarised the key takeaways at the end of day, taking on some responsibility for connecting the disparate exercises and readings to each other, instead of leaving that up to us. I think the idea to have a day about abstractions could be good, but this curriculum felt hazy and undirected. It was as if someone with a proper plan to teach us got sick, and a substitute threw together a plan at short notice, without more of a goal than to keep us nominally occupied.
A final afternoon session bringing everything together would be nice.
Optional readings at the end were too hard for allotted time

Agent Foundations 6 runs 137 responses, 175 written

for pasting into Claude

October 2026 Tue 27 Oct Aden Power 26 responses energy 3.5/5

Rated against what each person expected. 0 is as expected.

Overall +0.25 better than expected

Content +0.25 better than expected

Clarity +0.17 as expected

Motivation +0.08 as expected

Exercises -0.22 as expected

Readings -0.36 below expected

Other activities +0.10 as expected

Pace +0.04 about right

Difficulty -0.08 about right

Novelty +0.75 mostly new

Read what they wrote22

Worked well6

The conversations about Agent foundations and where it stood in relation to the rest of the field were very enlightening. Aden does a really good job explaining.
The discussion part
Aden's best day! loved all of the content he presented
I really like Aden's pedagogical style.
The toy models/games given during the lectures really helped illustrate concepts around agents
Aden's enthusiasm for the topics is contagious. Even those topics that I came in somewhat uninspired about, he helped me stay motivated.

Would improve11

Very good presentation, can be more structured.
I wouldve liked this day to be a bit more focused on the different research agendas, it felt a bit rushed as a whole. All the topics where interesting, i just wouldve liked more time on the foundations.
A lot of terminology was introduced for intuitive concepts which eventually became hard to remember and follow. Not sure if there is a way to reduce that, but that would have helped.
I really enjoyed the conversations we had about modeling things as agents versus actually beleiving they are agents. It would have been nice if we could have had that conversation a bit longer.
There is not enough time for the exercises. For me, personally, it would for example be more useful to have discussion time, explaining topics we just learned to one another.
this was super great! the guest lecture was so so (too slow to build the definitions, and not enough time to talk through the actual result and relevance of it) (that's not why all my scores are not max)
I would like to see a clearer connection between existing theory and practice if possible.
I did not like the guest lecture. Getting bogged down in details before explaining what it matters. Foundations need solutions on lower philosophical level, not more formulas. Also, first lecture didn't seem to know what it's talking about in the end. Lob theorem is not a "subfield" of foundations, and tiling is not inextricably tied to this theorem.
I wish this day were earlier in the intensive, particularly before multi-agent day. I think it also would have been good to clarify more why we care so much about copies and how we think about copies.
The lectures are good, just like yesterday. A lot of materials feel too theoretical in nature, and it would be nice to have hands on coding session to validate the concepts we learn and to see if they are really useful in practice
i think the guest lecture didnt really match the flow of the rest of the day and threw everything off

Changed my mind4

The embedded agents framework seems like a promising research idea. I didn't realize it was so underdeveloped.
I especially liked the final part, and I learned that some deep philosophical thought experiments have now become real-life problems, e.g. issues with perfect copies of agents.
Agent foundations is quite close to part of my research and the call of action for embedded agency was quite motivating
I felt like I gained a greater appreciation for why philosophy of the self or of the mind are important in AI Safety and reasoning about models.

Other thoughts1

i think it would have been good to have a real technical day on reinforcement learning and how it applies to LLMs in particular (exploration and gradient hacking etc)

September 2026 Thu 24 Sep Aden Power 28 responses energy 3.8/5

Rated against what each person expected. 0 is as expected.

Overall +0.50 better than expected

Content +0.35 better than expected

Clarity +0.27 better than expected

Motivation +0.62 better than expected

Exercises -0.42 below expected

Readings +0.08 as expected

Other activities +0.24 as expected

Pace -0.12 about right

Difficulty -0.31 too easy

Novelty +0.96 mostly new

Read what they wrote37

Worked well12

I bet the game would've been cool :(
Emphasis on group discussion and activities.
Activities and general pace were great! Loved them.
Aden is a charismatic talker and follows a clear line of reasoning/motivation in his teaching. Very good.
I think Aden's lecturing was again great.
- Seeing several of these core ideas/ agendas side by side made it clearer what each one is trying to formalise and how they relate
The game at the end was fun- something different!
Aden's explanations of Löb's theorem and embedded agency were fantastic.
Lecturer was very good.
Enthusiasm, explaining the different research agendas
The presentations were very clear and well structured.
I did like Aden's enthusiasm! Also the freedom to choose what paper to read was very good, as well as exercises for each of the 5 research directions. The game was a fun activity as well.

Would improve15

Today's exercises used a lot of notation to describe relatively straightforward concepts. The problems themselves also weren't as focused on teasing out the core ideas as I might've hoped. For example, proving that a binary search converges didn't add much to my understanding of why any agent that can respond appropriately to certain changes in the data distribution must've learned an implicit causal model.
Maybe some preliminary reading would have helped.
The reading for garrabrant induction was great but the exercise was bad and you should have put in the effort to have one awake and focussed human read it
Overall, everything was great. It's definitely a bit dense in material, and I'm not sure I got every detail, but as a high level overview of the very broad Agent Foundations agenda, the day appeared to be great. A bit meta, but over the course, I get progressively convinced that people need some external organization and splitting people into separate rooms with whiteboards and a list of topics that are connected to read material would be better than allowing them to self-organize or work alone. At least finding such rooms and encouraging everyone to use them (groups of 5-10(max.) people would be great).
I think more time spent on the mathematics of agent foundations, especially in the lectures, would have been good.
I think the exercises were interesting. I would have liked to have time to work on other exercises from the other proposed topics. Today's lectures were refreshing with respect to other days, I think I enjoyed it more than most of other sessions.
- A lot of ideas in one day! (coherence theorems, Löb's theorem and tiling agents, logical induction, decision theory). Going deeper on two or three would help - A short refresher on the formal logic before Löb's theorem, for people without a logic background
A bit of a muddled up day. Some parts were good but it was a bit all over the place, it seemed like three lectures in one. Everything was claude generated as well- this is a pity and I actually quite dislike it.
The exercises were a littel hard. More concrete feedback on the specific feedback form
Nothing, I enjoy the day throughout
TV screen for slides is too small to see (23 curtain screen was larger); hot water machine broken (no hot water).
It was nice in general!
I thought today was very good.
I would add a coding demo for one of the topics.
I don't know if it was just me, but Aden might be too loud at times. But it might've also meant that my sensitivity spiked. When it was the last lecture on cooperation, I think clarifying that we do not consider empirical results or actually mentioning them for completeness (such that in a Prisoner Dilemma tournament tit-for-tat strategy, which is cooperative by default, won) would help so that no one else raises it or points it out. The day felt very abstract, which might be the field's feature rather than the lecturer's. The game was lacking in interpretability a bit. Discussing it in more depth would've been cool. The first lecture started very strong and I was so engaged, but once we started proving a theorem rather than just going through the main statements (proof sketch), I was lost -- for me it was engagement breaker. THe first lecture was also way too long. If I remember correctly, it lasted for 1.5 hours non-stop.

Changed my mind8

I have less confidence that the entire field is on solid ground.
In fact, I was more or less interested in game theory for quite a long time, but struggled to find time to dig deeper. Although I think that theories that treat people as perfectly rational agents are confusing, they must work much better for future LLM agents, even not superintelligent once, and having a good theory of safe improvements could allow to handle more problems and negotiate on a wider range of topic successfully. Besides, advancements in game theory have presumably saved lots of lives before, and allowed politicians and stakeholders to build better models for negotiations. I tend to believe that it might be one of the most promising research agendas in the nearby future.
Ironically, I am more and more skeptical of theoretical frameworks as the intensive progresses. I came in with a lot of enthusiasm for theories (I come from SLT and was expected similar things) but now I am more and more convinced that the way forward in patchy work like mech interp.
Much clearer perspective on safe self-modification and Vingean reflection.
Discovered applications of Game theory for agents foundations, so I could use my GT background in this direction
n/a
Decision theory is quite nuanced
That mathematicians and whoever works on all of this are "the drunkards searching for keys under the lightpost cause that's where the light is." It seems like once upon a time, a mathematician relaxed the infinite compute assumption and AIXI immediately manifested and killed them. Why is there so little work on inconsistent preferences and arguments beyond Dutch Books and Money Pumps? It honestly makes me furious -- no one money pumps anyone, and the theoretical possibility does not mean that someone will appear and do that, as that requires trying to find the inconsistency in another agent's preferences and then make sure that the agent would not notice exploitation (and he almost surely will, as he is not stupid despite inconsistent preferences). Logical Induction was the only promising thing out of everything taught today. No one maximizes any utility function, that's a myth, and for that to be happening agents have to have infinite compute / time and that is never the case. No human does that, even if they claim they do, because there is no way they can execute those operations on made-up probabilities and payoffs that do not take a lot of things into account. Aurelio raised nice criticism which was similar in spirit, and it was nice of Aden to respond by two arguments: 1) We study this so we do not reinvent the wheel, as oftentimes people wonder "what if the agents works like this?" and he goes "well, we already thought about this" and 2) So that we can understand the future machine that we will build. But argument 1 is weak because you can just tell people that their assumption is unrealistic instead of launching an entire research program preemptively, while argument 2 precludes us being killed by an agent with inconsistent preferences before we get to build a consistent one. I believe the field should abandon all the convenient math and introduce realistic assumptions (bounded compute, polynomial time etc) and work with them. Otherwise it's waste of math.

Other thoughts2

Overall great day.
n/a

September 2026 Wed 30 Sep Ashe Vazquez Nuñez 27 responses energy 3.5/5

Rated against what each person expected. 0 is as expected.

Overall -0.20 as expected

Content -0.12 as expected

Clarity -0.04 as expected

Motivation -0.36 below expected

Exercises -0.25 below expected

Readings +0.17 as expected

Other activities -0.05 as expected

Pace -0.28 too slow

Difficulty -0.08 about right

Novelty +0.92 mostly new

Read what they wrote35

Worked well8

Ashe is very enthusiastic and thoughtful, and engaging. That helped a lot.
I think I had high expectations which weren't fully realised, but the day was still good overall. I particularly liked the 3-person discussion.
I liked the philosophical discussion at the beginning of the day
The readings and the free flowing discussions.
Intuition for updateless decision theory and reflection.
I read about open-source game theory, which was interesting.
The speaker was clear
i liked the reading about logical induction

Would improve20

I liked this approach very much- I appreciate that it has tried to link it back to the content covered during other days.
Maybe it should be more broad and more impartial as in less lesswrong-coded, there are a few research directions that fit "agent foundations" idea well, but don't have a corresponding lesswrong post. As Iliad reaches out to people from academia and other backgrounds, it would be nice to accommodate them as well. Know thy audience.
Maybe a bit more of an overview of agendas would be nice, some broader classification perhaps, with blindspots. As the day seemed to be mostly driven by ourselves and readings. Also, the proof in the end pertaining to Intrapersonal game theory took much more time than was available to it, that needs to be fixed.
I see the value in this day, but it's hard to engage with these topics for three days (built upon earlier agent foundations that were similarly philosophical and qualitative). In particular, it is unsatisfying (as part of a 1-month intense programme) to have this many days at that level of intensity (after all, philosophical discussions are tiring) but not have some measurable progress. On the other hand, there were topics like steganography and data attribution that needed more than one or half a day to get a good handle of the mathematics upon which these fields are built. I would have liked to get a better feel for such topics, while contemplating the broader context of the work they entail in the background (as is common when carrying out any scientific research).
The come-up-with-a-project period was fun, but maybe not optimally structured for getting the most out of it. I spent a lot of time skimming various papers and aimlessly musing, didn't get very deep into anything.
Nothing, it was a great day. Perhaps, a more thorough introduction to the various research agendas would have helped.
As usual, I did not have enough time for the reading parts.
there was so much freedom. it would have been great to have a more directed teaching and program. The topics where sparse
It felt a bit hand wavey at times
I thought the day could've had more structure. The readings were very good but I didn't quite tie together for me.
Maybe would be better to bound more the agenda activity. The goal of the activity EAS not very concrete. Having discussions with other participants was great. It was less intensive than other days.
More time on logical induction trade-offs.
It felt like there was very little teaching today. I didn't find the overview at the top of the day to be very useful. The readings were fine, but reading can be done anywhere at any time.
This day could have had more time, especially given that we had three days for agent foundations, which was far more philosophical rather than quantitative.
In my opinion today needed more teaching and less reading / project development time. Most of us are seeing this stuff for the first time, so I think it's ambitious to expect us to be able to develop theories in new and interesting ways - we first need to learn and understand the material better.
I did not find the final talk was very pedagogical. It needs to be modified to make more pedagogical sense. It was too narrow. The "make your own agenda" is hard with the time we had. Agent foundations days should be back to back. It's the only way to make the "make you own agenda" activity make sense.
I found it too philosophical and not formal enough
It seems the day was little empty and lacked taught material or exercises. Given that the first day on Agent foundations was mostly theoretical, it'd be nice to derive some properties and maybe even see some models / dynamics in computational scenarios that can't be derived with pen and paper. However, maybe accounting for the fact most people are tired is also a good strategy.
actuall following in schedule more strictly and bridging the gap between very general statements and concrete theorems that we can propose and prove about agent foundations - in agent foundations day one we already went through readings with fairly high level of abstraction, so i think more concreteness could help.
I was a bit happy about "make your own agenda" as an exercise. The reason is probably that I am unhappy about the pre-paradigmatic state of AI alignment, where there's no established research directions (with established results and conjectures, senior and junior researchers, and open questions), but instead where everyone cooks his own soup and there's little feedback mechanisms and a lot of re-inventing the wheel. Maybe the exercise could have talked about this? This might also not be super specific to today, even though today's exercise triggered it.

Changed my mind5

Decision theories are much more fundamental to safety than expected.
It's interesting that MAISI is working on open-source game theory.
n/a
I updated my perspective on Open-source game theory, and I'm increasingly convinced it's one of the most important research directions (or maybe some of its extensions or modifications) to pursue. It won't solve alignment, but will definitely help in international agreements, illustrative examples and building the foundations for the agentic economy working at superhuman speed (the current one is kinda broken)
time is an important dimension to think about because not all timescales are equal for all agents

Other thoughts2

Superb continuation of agent foundations.
n/a

August 2026 Tue 25 Aug Aden Power 29 responses energy 3.5/5

Rated against what each person expected. 0 is as expected.

Overall +0.62 better than expected

Content +0.55 better than expected

Clarity +0.72 better than expected

Motivation +0.34 better than expected

Exercises +0.33 better than expected

Readings +0.56 better than expected

Other activities +0.45 better than expected

Pace -0.14 about right

Difficulty -0.14 about right

Novelty +1.03 mostly new

Read what they wrote32

Worked well8

The game theoretic part was super interesting, thank you for allowing so much time for discussions during the lecture! Also the game was super fun!
Great teacher
Aden is an incredible teacher, I really enjoyed today!
Aden put lots of time into making slides, materials, lists or readings and exercises for every topic!
The game at the end was really fun, it was cool to see an actual game with an evolving optimal strategy in practice
Liked the readings with multiple choices.
The game was fun and engaging.
The logical induction mini track!

Would improve13

Just do more days on Agent Foundations. It is too much content for a single day. In general, it was nice and less hand wavier than other days. I feel like with a single day it's hard to do better. Again my general recommendation of "just let us do lecture notes" during the whole day and be available to help applies (at least to me)
Would've wanted more discussion about actual applications of this to safety.
People spent a lot of time being confused on definitions, maybe keep important definitions on all relevant slides.
Great teaching! Just confused about the agent foundation field, it all seems very abstract and it is unclear what the point is to me
Aden is such a good teacher that I'd rather have him teach than me read or do exercises.
would be interesting to know some main result in lecture from each of the subareas. still not very sure if I should spend time in this area.
More concrete examples, especially for embedded agent. I’m still trying to think of a concrete example of where dual agent theory can’t explain something that embedded agent theory can explain. A concrete example of the game we played at the end before it started.
group exercises
More structured overview of subfields of Agent Foundations, since it is not really a single field like the other days. (If at all it is must be a single day, participants should have a way to go much more into depth into one approach comparable to other days, rather than just a shallow overview of many topics. There should be mentors knowledgeable about each specific subfield so the participant can learn from a person (which is much of the point of ILIAD Intensive over just reading articles and papers online), and which mentors are knowledgeable about what should be broadcasted, loudly.)
setup was really good! lectures were quite engaging and I think agent foundations as a field probably benefits from not forcing depth of formalism in a survey where it might struggle to improve understanding
I think I would rather prefer a Reinforcement learning day to be honest.
Already gave feedback individually. Might be worthwhile to checkout Michael Dennis' PhD Thesis for the self-referential claim. Slides 14 and 15 are quite helpful and some of the best part of the day. Seems like I'm super biased towards today materials as I'm actively working on it. Aden is a pretty amazing teacher. I also really like the approach of learn whichever way that's best for you.
It would have been helpful to cover more mini tracks within agent foundations in more technical detail (e.g. AiXi, natural latents etc.), but I assume this would take more than one day?

Changed my mind8

Agent foundations seems boring and very formal
not sure about agent foundations, will have to look into each subarea still
I feel agent foundations might be a new interest for me and I am glad I was introduced to it :)
Agent foundations is better than I thought.
it is worthwhile to create formalisms for analyzing possible scenarios for ai safety
I’m gonna pay more attention to open source game theory now
I liked Löbs theorem!
I'm starting to wonder that alignment and safety is not something that we can fully solved and we have to keep working on it even we are in a short timeline take off scenario.

Other thoughts3

The game was very fun.
N/A
Seamus and Pablo raised a really good point that agent foundation work in the opposite way of typical scientific discovery. Typically in Physics, you solve some specialized thing first then generalized. So I'm uncertain about the agenda. Nonetheless, I don't think I can find myself working on anything else but agent foundation since I think it's the work that is most intellectually satisfying.

June 2026 Thu 25 Jun Sung En (Daniel) Chiang 12 responses energy 3.1/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.4 of 5

Teaching and format 2.8 of 5

Difficulty match 3.2 of 5

Energy 3.1 of 5

Read what they wrote13

In their own words11

I did't like the topic, found it unreasonable and poorly structured.
I think that the slides during lecture had too much text. It was hard to go over all of this content, and at the same time listen to the speaker.
- I think I got most out of the lecture - The readings were generally interesting - The cross-pollination prompt for Thermodynamics x Descriptive Agent Foundations wasn't very useful; and we didn't manage to come up with any interesting connections - The descriptive agent foundations ideas were interesting, but some parts like the type signature (A -> B) -> B we didn't really understand very well - Maybe an improvement would be to have the teachers move around and come to every discussion group to answer questions/help spark discussion - Better cross-pollination prompts would also help - The morning lecture was way too fast, specially taking into account that some concepts are quite complicated - Specific examples accompanying the explanations during the morning lecture would have been useful - It would have been useful to split the morning lecture into two pieces with a break in the middle
I found the concepts learned today interesting, especially the Gödel-like Löb theorem, while the relevancy of at least some aspects of agent foundations to practical AI safety can sometimes be questioned (it does not seem the frontier labs care about proving their AIs consistent, feels more GOFAI). The lecture this morning was a bit dense, the 2h could have maybe been split up into two sessions. Also, the slides were a bit text-heavy, with full sentences on the slides fighting for attention with the speaker. Reading material and discussions were interesting and insightful.
I really liked the content. Agent foundations are a fascinating topic and the reading lists brought together what I imagine to be a very informative subset of the core articles on the subject. I liked that we got time to discuss with people who had done other readings. Maybe the cross pollination prompts could've been more extensive as in our group we found it difficult to pin down the connection between the Thermodynamics x Optimization ideas and Descriptive Agents. Overall I think it was a fairly informative day. One more suggestion is that maybe the lecture in the morning could've been more engaging as after a time it felt like the lecturer was speaking into a void even though as far as I can tell most people were listening attentively (based on how people related things mentioned in the lecture to discussions in the afternoon). I should also mention that while the readings were quite good today, perhaps there was too much material that was kept as pre-reading/prep. Compared to some of the other days (like prosaic alignment or Mech Interp), Agent Foundations is precisely the kind of area where we would have benefited from having more discussions and/or lectures (even short 20 min ones) on some of these highly technical topics.
I do think the content has a place, but the point of the course should be to teach and engage. I think these quite subtle ideas should not be done in handwavy machine gun fashion.
I liked the notes i read in advance ("optimization and thermodynamics"). I think the lecture in the morning might have been more effective if there was less text in a larger, more readable font on each slide, and the speaker sounded rushed; I think maybe these are both indicators that the depth might have been to ambitious to jam into one lecture. since there was a ton of overlap between the lecture nd readings, I don't think that level of detail was necessary.
-I understand that today was supposed to be a survey, but I think it would have been better to really develop a few compelling ideas rather than many diffuse ones. -I think the lecture slides had too much text to be extremely useful for anyone but the presenter. -The extra resources are nice. I think they could be used in a more primary way as they are more formal. -The optimization and thermodynamics stuff is cool! -Also, a general thing I've noticed is that using LessWrong articles seems to undermine the credibility of the content for some participants. This is particularly damning when the content itself is already quite speculative/fuzzy.
The slides were too small, too much text, and the presentation wasn't engaging. The content is good, though. The format of breaking out into discussion groups based on the readings is good in principle, but the discipline to actually do the work was somewhat lacking. Overall, was a good day and the readings were well selected.
I wasn't able to understand what this was about.
Very interesting formulation of the day, and good discussions with everyone about where the field is in the broader context and where AI alignment is moving. Learnt a lot.

Other thoughts2

I look forward to the colder temperatures next week as heat-induced sleep deprivation severely limits my capacity atm.
descriptive agent foundations / selection theorems connect directly to natural latents, would like more of that thread.

April 2026 Sun 19 Apr Sung En (Daniel) Chiang 15 responses energy 3.6/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.4 of 5

Teaching and format 3.7 of 5

Difficulty match 3.2 of 5

Energy 3.6 of 5

Read what they wrote36

Worked well12

Interesting discussions and readings
The format was good (and I enjoyed the forcing of small group discussions from people who read different topics), and the short lectures from different researchers was interesting.
The pre-reading combined with group sessions lead to interesting discussions. But I am not advocating for having pre-reading for all modules, that would be too much.
The option to chat in groups was great would have liked more time for that.
Guest lecturers were good, cross pollination is a really nice format that people seemed to benefit from. The exercises have so many valuable remarks they are close to supplementary lecture notes.
teaching format, early-sent materials, guest lectures, and a nice mix of lecture, exercises, discussions, and guest lecture makes today feel much better and more balanced.
The private small groups with advance reading were great! More like that, please! I think because we’ve known and liked Daniel for two weeks, I felt invested in wanting the day to go well for him, so I even if I was tired, some subconscious relational imperative kept me more engaged with him than the one-day teachers?
Talk with Aram was nice, group discussion worked reasonably well
I think the topic selection was important, i had been aware of some agent foundations concepts and also the somewhat dismissive attitude towards the area. (I got the impression from some participants that they've come to share that sentiment to some extent.) My appreciation of the questions increased and i at least understood how they relate to each other and AIS in general.
The mixture of lecture, exercises, readings, and discussions was nice. I enjoyed having different formats to learn that switched about every 90 minutes and it kept me more engaged.
Thorough introduction to the basic concepts, explored by discussions.
The discussions were really good, I enjoyed the conceptual nature of the lectures, with the more theoretical maths based exercises.

What they took away12

Decision theories
Thinking of mutual information between an agent and system as a resource for optimization was an interesting perspective.
I now have a better sense of all the modelling difficulties introduced by embedding agents in a larger world.
The lecture about studying the physical origins of causality and choice was extremely useful and insightful.
Premise of algorithmic thermodynamics
decision theory
Newcomb’s paradox splitting the world into one-boxers and both-boxers, with both groups thinking their position is obvious!
CFT vs FDT idea (and from non-normative perspective), embedded agency idea
That the confusion and problems that i've struggled with in connection to the arrow of time, causality, entropy etc. are shared and not understood with a consensus on the wide AF community. It gave me hopes that i have a specific contribution to make currently.
Agents are part of the world rather than having an entire understanding of it.
Open problems in decision theory and how they may practically relate to AGI.
Recapped on different reward modelling systems. Learnt about lobs theorem (and the theorem it's based on)

Would improve11

I found the talk by Cole not great, I felt he was a bit rambling and it was hard to follow. That's a comment about the presentation style, not about the content (which I think was good).
I think I'm still a bit skeptical that some of the more abstract topics we've covered in Illiad will be concretely useful for AI safety on reasonable timescales, but perhaps this is more personal taste than fundamental objections.
More realistic scope/focus
The first two lectures could have done with a break. The second lecture also had some unnecessary overlap with the first and the idealized agency module.
The lecture about AIXI really wasn’t useful to me and seemed to not really be an improvement. I would have preferred a lecture on one of the other areas of agent foundations. I liked Daniel’s lecture but the text on the slides was too dense.
So much content a lot of other people didn’t get to the exercises
please don't order fast food anymore, instead can we have more Asian food please please!! Also can we also go to Asian restaurants after class as well. I could barely ate anything yesterday and today due to how the food choice is.
Daniel seemed nervous. He could have talked slower. The fundamental reading was pretty fully redundant with his talk? I didn’t see the need for both of those. (The pre-reading was non-redundant and valuable.) Cole Wyeth’s talk was perplexingly bad. If he was just extremely nervous (to the point of tripping over his words and repeating himself so much that he ran out time for any non-redundant material before the end of his hour-long slot), I would like to extend my sympathy and hope that getting practice helps him feel better over time. If he just wasn’t unprepared, I wonder if we could tactfully require that he rehearse before coming back.
I think the initial lecture and Cole’s lecture could have been more focused and clear in scope - I found I didn’t understand that much of the first because it was perhaps too ambitious in scope, and I would have learnt more clarity from a more restricted and focused session, and I think somewhat similarly for Cole’s lecture, I found it somewhat unfocused so struggled to get clear points out of it
In the material: the acknowledged existing challenge to reconcile many opinionated and partly contradictory research agendas may be the root cause of some unfavorable views on the field, and a more positive outlook could be conveyed to everyone with evidence that "integration to real science" is ongoing. In the teaching format: I felt that the afternoon was divided into too many segments and assigned discussion/exercises did not get enough dedicated time to work on them.
I would add some content on "where are we today" to explain where the most modern agents research is and what it can currently do.

Other thoughts1

Generally it would be good to get a list of "what we should take away from this day/content" at the end of the day, or a set of questions to answer to test what we have gotten from it. This is not specific feedback to this day, but more in general for the course!

AI Alignment Introduction 5 runs 132 responses, 237 written

for pasting into Claude

October 2026 Tue 6 Oct Margot Stakenborg 37 responses energy 4.0/5

Rated against what each person expected. 0 is as expected.

Overall +0.57 better than expected

Content +0.54 better than expected

Clarity +0.57 better than expected

Motivation +0.59 better than expected

Exercises -0.20 as expected

Readings +0.22 as expected

Other activities +0.54 better than expected

Pace -0.11 about right

Difficulty -0.30 too easy

Novelty +0.11 an even mix

Read what they wrote59

Worked well21

margot is an excellent speaker and the ideas were well communicated
The presentation and discussion covered a wide variety of topics.
The discussions were very fun!
The discussion sessions.
The discussions about the readings were edifying. I learned a lot about the history of thought in the AI safety field
The group discussions after each reading
The discussions
Discussion periods and socratic method.
I thought the conversations were thought provoking. I got a good idea of the various philosophies around alignment and the open questions people generally struggle with.
Margot teached super well!
Margot's design and delivery and facilitation of discussion was excellent.
Lot's of discussions
The readings given are nice and we got to discuss them
extremely interesting topics with great explainations.
time splitting - the decomposition into various parts such as lecture, reading, discussion was well proportioned
Really excellent and enthusiastic instruction. Margot's depth of knowledge and ability to communicate are inspirational. I am meaningfully changed for having attended yesterday.
Peer discussion
reading and discussion sessions were good. It gave to chance to read at our on pace about things that interest us. Discussions gave an overall impression of all other topics.
For this flavor of non-technical content, the alternating reading/small-group/whole-group discussion format was great!
Margot's lectures were very nice and allow us to gain a clearer picture of the field and a better understanding of the jargon employed in AI safety. It was very nice to go through the details of the Hugging Face incident. Multi-agent interactions played a crucial role in this context, and this motivated me to develop a theoretical framework to investigate interactions between different agents using Partial Information Decomposition. I already had some ideas about this before the intensive, but this lecture helped me arrive at a clearer picture.
passionate, clear speaking, extremely knowledgable and eloquent speaker!!

Would improve20

A bit more silence during the reading sessions to increase concentration; the amount of reading was too much, and the continuous talking during the reading sessions was not helpful. Also, the lecturer was highly knowledgeable but it would be better if she just spoke a little slower and took some time to breath :)
i liked the quiz maybe we can more/harder quizzes that force us to solve some small hypothetical problems
Today was a great day! One thing that was a bit tricky for today was that we were asked to do a bunch of reading, but quickly the instructor would engage in interesting but possibly slightly off topic conversations which made it a bit difficult to concentrate on reading.
A lot of the discussion veered quite far from the articles in the slides. Unsure if this was intended but a bit more organization would have been helpful.
The discussion needs to be more structured, if we have 25 minutes to go through some posts and discuss them in groups then maybe we ask all groups to post a brief summary 2-5 lines on a forum which is a link everyone else can access. That way so instead of asking each group to speak out what they discussed turn by turn, we just take 5 more minutes to read the forum. Time management seems to be something that can be worked on the above is just one suggestion in this vein.
Was very good, maybe some more structured question sections?
The questions after these do not ask what my original expectations are and thus contain no information about the quality of the presentation
A lot of the reading was hard to parse and used a lot of unnecessary jargon
Maybe having less time to read the paper and more time to discuss.
More concrete examples and connection to the underlying theory of LLMs, perhaps a step-by-step walkthrough of how particular cases of misalignment might work given what's been covered in the previous intro to ML Engineering class.
Reading structure
I really enjoyed all the discussions. From what I understand the course will not include that much more of this broad overview of alignment philosophy out there and I would just like to hear more about this.
I'd go more into detail on the readings before getting us to read them. Or at least, I feel like today felt quit slow/uneventful as a result of so many 45min reading breaks we had.
Encourage the readings more. For instance, encourage people to go to a quiet place to read, and take discussions to another room.
More concrete examples of alignments vs misalignments. Motivate with important examples/case studies before introducing them.
this was a great session, but I would have liked to do it after having acquired more technical knowledge on the models and the training process.
It was great a overview, slightly more quantification would have been nice, e.g., real data to and consequently discuss texts like AI 2027 and Europe 2031 from another critical angle
More time for readings
It is a bit difficult for me to read papers live and then try to comment on them. With only 25 minutes to spend on each of them, I can never get too far into them.
this was amazing! i appreciated the reading+discussing format, found it quite distracting that the speaker would answer attendee questions during the reading time! now i need to chose between reading the material and listening to the very knowledgable clear and passionate speaker. and if i try to read, i am constantly distracted

Changed my mind13

I feel a bit sorry for the security people at OpenAI after reading the long Tweet.
yes i realized why AI safety research is important and how sometimes you can take a step back from doing math and think about why are we doing this
Yes, but the exact topics elude me at the moment.
More concerned about ai risk
I didn’t change my mind on any subjects today
Started having an interest in the economics of AI agents
Not really.
It does seem like there are predictions that the rationalists (pre-LLM) got wrong. I am interested to hear more about this and will follow up on it.
I would have probably kept this section for next week or the week after
The possibilities of large AI models misaligning is very high
I am slightly more sympathetic to controlling AI and/or slowing its progress than before.
I was not aware of gradual disempowerment until today, which very pointedly describes a very reasonable concern one could have even in the more conservative cases of capability growth.
i found

Other thoughts5

N/a
Nothing
I did not like "from each table a person speaks vaguely about a vague preceding discussion" format. The readings were not matched against the allotted time. I barely got to the point of long papers I had 25 minutes to read -- and then there was no concrete discussion of that point, so I had to talk to ChatGPT to get at least some well-formed takeaway.
Nothing bad about the lecture, it was just the content was too philosophical to follow for me personally especially when I was suffering from the jet lag.
can we keep going until 6pm on future days?

September 2026 Tue 22 Sep Aden Power 36 responses energy 3.8/5

Rated against what each person expected. 0 is as expected.

Overall +0.77 better than expected

Content +0.89 better than expected

Clarity +0.91 better than expected

Motivation +1.14 better than expected

Exercises +0.30 better than expected

Readings +0.29 better than expected

Other activities +0.34 better than expected

Pace +0.11 about right

Difficulty -0.60 too easy

Novelty +0.17 an even mix

Read what they wrote58

Worked well20

I thought Aden's lecturing was great. I thought his explanations were very clear.
The suggested readings were engaging, but I would have preferred more time (eg by providing references in advance).
I enjoyed thinking about broad alignment and safety questions - it was a welcome change bringing alignment and safety front and center of the materials again
The sense of humor of the lecturer!
This talked felt very relevant for todays issues in AI safety. It would be a very different talk 6 months ago which is a good thing.
- The readings mixed classic papers (CEV, corrigibility) with current material (Claude's constitution) - The city walk
the pace of change between lectures, reading and discussion was quite fast, but I think that ultimately it's a good thing
It was a good overview of the field of alignment and introduced a lot of new ideas.
The presenter is good. The technicalities are weak.
Speaker was very good
the overall content feels very relavant to the topic of AI alignment and covers argument from both sides and rather than presenting one point of view as superior than others the points of view were all given in a fairly objective manner.
I enjoyed that the lectures strived to give the big picture instead of focussing on introductory technical details. The overview is instructive
The level and the discussions
discussions were informative
The topic itself was very interested and so was the discussion.
Best day so far. Was great to finally hear about alignment and to have discussions about perspectives on alignment. Aden was a very engaging speaker.
Aden is fantastic and his energy is very infectious. Today really helped give a bit more context too the technical stuff we did before and was very refreshing
Clear taxonomy of alignment agendas and London tour.
The lecturer is very energetic and did a very good job of motivating the problems/views, which is the main objective of this lecture.
I think the presentation of the material was great, with a good mixture of seriousness and levity. It was very evident that Aden had thoughth a lot about these things and could give general answers and a variety of perspectives on all of the (sometimes controversial) topics.

Would improve25

There was not enough time to read the suggested readings.
I think the timings? Somehow it seemed like we didn't have very much time for readings and discussion - for a few of them, I only managed to read a few pages before it was already time for lectures.
More time for reading, either by giving less reading or providing it in advance.
1. Have this day earlier in the program, 2.have more time for reading
This was quite good.
- Felt compressed as a half day, and could come earlier in the course since it frames everything else - More time on critiques of the outer / inner alignment split
On a philosophically oriented day, I think this was as good as it could be.
I think it was a great day overall. I loved that the day was discussion focused- having well-formed opinions on things like risks and solutions can only come by discussing and stress-testing these opinions with other people. There should be more of this!
everything good, that should be moved to an earlier point in the intensive
Nothing. The day was excellent.
I thought today was very good and presented material in a motivated fashion.
Was good enough today
Most of the staff was really good. Sometimes it was difficult to keep up with each reading session, but I suppose that's related to the short day. We still didn't fully connect it to any of the previous days. Some connections are easy to make, but the most interesting input from the lecturers could cover some non-trivial connections, that aren't immediately obvious.
Minor quibbles on the slides: I don't think the definition of Inside vs Outside View was the same as the standard usage, the hand-wavy equation used to explain Inductive Bias was perhaps fuzzy enough to be pedagogically anti-helpful. On the big-group discussions: I liked that there was lots of back-and-forth during lecture. However, a good chunk of it seemed to quickly lose sight of the topic at hand. I felt like some fraction of the things people said were either (a) intended part of a different, non-alignment-related conversation or (b) so vaguely phrased that I couldn't understand what was meant or what role the statement was meant to serve in the discussion (Observation? Rebuttal? Thesis?). Perhaps it's possible to find a way to guide, shape, and clarify discussion without stifling it? I don't know of any magic bullet for making this happen though. On small-group discussions: Seemed like people mostly didn't finish the readings with enough spare time for a good discussion.
I thought this day was run particularly well.
I think today was nice in general. Maybe a bit more of time for reading on our own.
Clickable links in the notes to the different articles that are mentioned? Save lecture notes in a different format than the beamer so it's more pleasant to take notes and follow along.
Nothing. I really liked it today
Best day so far. Was great to finally hear about alignment and to have discussions about perspectives on alignment. Would've been nice if this day came in week 1
Could've been a bit more time for the reading
More time to discuss threat models.
Having this as the first lesson in the curriculum!
Maybe a little more time for the reading could be good. I barely managed to get into the texts by the time we were asked to discuss, especially when I asked Aden something after the lecture (and before starting the readings).
the slides could have been a bit better, but the teaching was great
move this to week 1 day 1

Changed my mind10

I do not think an AI agent cannot be goal directed.
It is surprisingly hard to define alignment.
n/a
I need to read up more on how people think about alignment, not just about the concrete technical things being done to understand the capacity of the current models but about the what the future directions would look like, as an extrapolation from observations we've already made about the current models. because the current models will be the weakest the models will ever be..
Reasons for pessimism at least feel more salient today than they did yesterday, although this may be more a change of mood than of beliefs.
I gained some vocabularly for some of the more philosophical aspects of alignment.
Learned a lot about current state of the art and community beliefs about AI alignment.
I am still unsure if there is anything we can do at the moment, but the point that people in frontier labs are (perhaps)reading alignment research gave a bit of hope for the future
Sharper distinction between capability vs alignment targets.
Recently I have been more worried about the misuse aspects of AI, and today reinforced my concern about alignment, mostly because of the discussion around instrumental convergence.

Other thoughts3

n/a
I left after lunch because I wasn’t feeling well
Tour was a great break before proposals.

August 2026 Mon 10 Aug Leon Lang 33 responses energy 3.9/5

Called “AI Alignment” that cohort.

Rated against what each person expected. 0 is as expected.

Overall +0.03 as expected

Content +0.12 as expected

Clarity +0.52 better than expected

Motivation +0.61 better than expected

Exercises 0.00 as expected

Readings +0.03 as expected

Other activities +0.21 as expected

Pace -0.36 too slow

Difficulty -0.55 too easy

Novelty -0.33 mostly review

Read what they wrote72

Worked well17

Loved the discussion and sharing ideas.
I loved the group discussions :)
Discussion sessions and socialising exercises
The readings were really helpful and the questions were good too. They needed some form of thought provocation
met others, learned more about AI safety philosophy
The vibe and the plan on what to learn
Getting to know everyone well first
Discussions
The main lecture by Leon. I don’t think we can start the intensive without an overview like this one. At the same time, it feels pretty hard to deliver a lecture like this one in any finite amount of time.
I did enjoy talking to other people and felt like this facilitated well!
I think the introduction part was long but good! Feel I got to meet the bulk of everyone and it made the discussions way less stressful.
Food was super great, there were similar sandwiches in iliad 2025, and those seemed the same, they’re really great Some of the readings I didn’t know before were nice, I wasn’t super familiar with CEV and now have at least a hunch (although this is something I would like having more time to actually read the paper)
The lecture and the reading groups!
The first half-day of just socialisation was very good. It was especially effective because of the rules to ensure people talked to many people in different formats instead of the same people over and over. It might have been better to give the meetings more time (say, 10 minutes each).
The level was exactly right for me
The discussion was helpful. We split the reading between the four of us and trade readings and summaries. Everyone describe the reading well as they are good explainers.
The lectures were actually very clear and easy to follow. Also the socialisation activity at the start of the day was really helpful.

Would improve27

getting people to form teams for certain tasks could be improved
Defining terms a bit more clearly
I felt like some of the reading and material was a bit vague, e.g the volition stuff.
Pretty satisfied with the material
The 1on1s were very intense. It would be nice to have some break inbetween; at one point I completely lost track of who is who and who I have talked to already
We spent too much time on the presentation
It would be nice to go faster/in a bit more depth about what’s close to the frontier in terms of research.
Things were ok but a little disorganised. Eg we went over time in the lecture and the discussion part felt quite rushed
A bigger hall maybe. And also the first session is a bit heavy on the philosophical side of things.
Less time spent just talking in the beginning, maybe some game or something? Less computer problems during the presentation.
I think the slides could have a more interesting narrative. They felt kind of piecemeal.
A more diverse set of activities to meet people beyond 1-on-1s. Maybe games, questions to answer, etc.
Dont make this a competition by telling people they get evaluated. Creates weird atmosphere. Also biases towards extrovert personality types.
too many iterations meeting people, four sets of readings and discussions was also too many iterations. would be better to rotate groups or ask to do readings beforehand
A bit less timewasting.
Have mentors occasionally supervise the discussions.
Organise a little bit better the 1-1 time so that there is more time for the discussion of the readings.
I feel like we got bogged down a bit in the slides and the 1-1 in the beginning seemed a bit long
For learning, I would consider making it more clear which things are definitions and which are more philosophical concepts. I understand the goal is to touch on both but some of the confusion was not being able to distinguish the two. Overall I think it was a great introduction and I found it very helpful!! :)
Group discussions are nice but it was a bit of blind leading the blind. Also 2.5 hrs for intros felt a little long. I think a bit more explicit stress in the lecture (like 5 mins) on why safety is actually hard (like why control is hard) would have been good for me to hear.
Readings were super cool, but we didn’t have time to read them. Discussions were not super productive, some people from my group didn’t seem to understand the concepts (for ex., inner misalignment needs some diggesting, concepts are mostly philosophical, so if you don’t take time to digest them, they will seem useless or not worth it or not well defined- which is what people from my group thought. It was hard trying to convince them this concepts are actually temptative ways to make our intuitious about alignment a little more rigorous in a more philosophical/conceptual way) Some improvement ideas: actually have a little more time to read (like the 20min suggested on the website, because since class was too long we only had 2h for the 4 sessions), maybe send this to us beforehand and say the readings are mandatory previous work (then if people don’t read that’s on them), have less time for the discussions maybe? I think since this iliad is already short, most of the more conceptual readings could’ve been homework and we could’ve done more cool stuff (for ex., RL day was cut short and I’d say it seems better than today?) Unrelated but the class is a little too full and with people coming in and out all the time it’s kinda noisy/bothering
It might be too early for this and probably it will come later on, but a mix of non-technical and technical readings would potentially be fun! It would also be helpful (but maybe unrealistic at this point) to have some more precise technical questions to think about in the background, while we learn the material.
The topics covered. Most of them felt like arbitrary categorisations - "definition tennis", so to speak. For example, today did not at all make clear why it matters whether a misbehaviour by a model is "reward gaming" or "goal misgeneralisation" - so why spend so much time learning about these definitions? It's possible that the distinction is quite relevant in certain kinds of AI safety research (for example, there may be an intuition that a certain AI safety measure can combat reward gaming but has no chance of preventing goal misgeneralisation), but then it would make for much better motivation to discuss the difference later when a motivating example is available. I feel this was also reflected in the questions and discussions of participants today. Most points felt like they were just about unimportant technicalities of the definitions - not through any fault of the participants, but because they were trying their best to take the content seriously and getting confused (as did I) about these definitions which are really attempts to cluster something with a fuzzy boundary.
Give more time for reading sessions, or focus the reading material more on the important/relevant ones; it felt a bit rushed, and I would have liked to dig a bit deeper into a selection of them
I strongly feel like the material should briefly cover multi-polar failures and pluralistic alignment. In my limited but somewhat informed opinion, this area of research is quite neglected and at least it is important to make sure that people who chose to work in alignment is at least aware of this failure mode.
I think it would've been better to compress today's content into the morning, and do something technical in the afternoon: - Most of us are presumably here because we've already read enough LW posts or similar to be convinced that AI alignment is important. - In the discussion phase, I was often asked to have opinions that I don't think I can intelligently form without having gone through the technical part of the course yet. - Given the compressed calendar, the choice isn't between "present X technical content now vs at a time when people are all present and rested and not jet-lagged; it's between present X technical content now vs not at all."
I would probably have structured the discussion a bit more, like maybe asking various groups to pick a topic, but to be honest the freeform discussion worked well

Changed my mind19

The importance of fundamental theory within AI research
Many different perspectives in alignment that I hadnt thought about much.
We did discuss newer forms of alignment beyond constitutional AI eg. ai as willing servants which I thought was fascinating.
Too early to say
AI Safety, is more philosophical than anything else
Little, since most was review of information I already knew.
Still deciding between ai safety and math academia. This was interesting info. I found having to explain and defend the intelligence ≠ goal directed post made me understand it much more.
No
understand more why certain ai safety agendas pursued
Alignment theory matters not just practice
Earlier in the day I thought we were doing too much social work, immediately changed my mind during lunch and the afternoon discussions
I gained language to describe what I’m looking for a bit better: Empirically guided theory work
It is too early to say anything life-changing happened but I definitely feel this did not scare me away from AI safety, which was a mild concern going in. The sociability and energy of the whole experience is very good too, could definitely see working with people like this!
I don’t think I changed much my mind
I learnt about some points that make AI safety more difficult than I would have thought (e.g. even AI system with harmless goals are not necessarily harmless)
I read some of the CEV paper by Yudkowsky, which made me think a lot about what "alignment" really means, particularly the "extrapolated" part and the "coherent" part (interpolating different people's values). I am now unsure whether there is really one correct answer to this question, as different humans could have values that are simply incompatible, and especially the concept of extrapolation could have different answers that different people might consider to be the correct answer (for example, I think it's a consistent and plausible opinion to hold that in most cases one should actually not extrapolate and operate by the current opinion only, though I don't personally hold this opinion).
The career prospects sounded way more promising than I anticipated
I'm more and more concerns about misused more so than misalignment by any parties in the society. I think catastrophic outcomes could come from both human initiate ones more so than misaligned systems.
To be honest not really, other than I think I now take the arguments on instrumental convergence a little more seriously than I did before. I very much enjoyed seeing the whole AI safety field laid out broadly rather than just alignment in this initial lecture

Other thoughts9

Wouldve been nice to have some more time to read.
No
Some feedback on discussions and exercises would help.
The readings were a bit too much to handle for the time allocated
Overall good a looking forward to the program. Responded were a bit negative due to not getting through much today.
It might be nice to have some socialisation with other people at Lighthaven or otherwise connected to the topics of the Intensive (of course, only if they are interested in or willing for it).
The presentation could have been a bit faster and it would have been beneficial to cut the emerging discussions a bit shorter, as this wasted quite some time with limited benefit for the whole cohort Also, a lot of time was spent on seemingly older concepts, which might be a bit dated? It would have been beneficial to make it more explicit what is actually still relevant today and what is currently the state of the art …
I think Leon did a great job of summarizing and explaining the very vague terms into graspable concepts with useful examples.
N/A

June 2026 Mon 8 Jun Leon Lang 12 responses energy 3.4/5

Called “AI Alignment: an introduction” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.7 of 5

Teaching and format 3.4 of 5

Difficulty match 2.5 of 5

Energy 3.4 of 5

Read what they wrote13

In their own words10

The general format was good. I found the second topic most interesting and would have liked more time to understand the results related to inductive biases.
Different details of alignment were somewhat tricky. I.e. I had trouble wrapping my head around GMG, as well as the motivation behind the training stories. The afternoon sessions were really nice and useful – people were quite engaged. The weather?
I enjoyed meeting many interesting people and getting my head int the AI alignment problem space. Points to improve: discussions could be planned better (who reads what, how to divide groups, which reading needs how much time), some texts chosen were low quality or simply too vague to be useful in my opinion; I would appreciate if we try to discuss basic assumptions that go into the arguments stated and prioritize texts that do include rigorous arguments as opposed to story telling.
I liked the leisurely pace for a first day, it was engaging without becoming overwhelming. I would like to have a glossary of terms, since it's all pretty new, and more structured exposition. I also would have liked a clearer statement of the context
Nice introduction to the main contents of the module. The 1-1's were also very stimulating, although I was not very familiar with some of the topics presented/discussed in the material. Yet, the discussions with the other participants helped me a lot clarify some concepts and technical notions.
general overview was good; the slides were clean and well-explained; the discussion groups were nice; the reading was a lot for the amount of time, but overall it still worked quite well. I was able to think this through more carefully
Time for reading was a bit short for the lengths of texts. Discussion rounds worked well, maybe brief summaries of what was discussed in each group could be useful to other groups.
First day, so mostly looking at readings and posts that most people familiar with AI Safety have already seen so that's why the rating for difficulty. 4 (and not 5) for Content and Format because it took some time to get started today as we didn't have our allocated space ready in the morning, and we shuffled the activities a bit.
- The lecture and the readings worked well - The 1-1s were very nice to meet other participants - I think the 1-1s should have had some bell ringing to inform everyone simultaneously that it was time to switch pair. In practice, people went over time and I found it hard to meet many different people. - I liked distributing readings - I think it would have been nice to be forced to change discussion groups more in the afternoon
Read-then-discuss small-group format worked well, the discussions were the most valuable part, more than the slides. Most useful for me was Session 2 (alignment-problem decompositions): emergent misalignment is my own research area (I ran a BlueDot EM sprint fine-tuning a 7B), so it was genuinely useful to pressure-test it with people coming from SLT and agent-foundations angles. Session 4's empirical-vs-theoretical framing also lined up well with my AI-control work. To improve: a lot of the Day-1 intro was review for those with prior alignment background (BlueDot or equivalent) - a fast-track option or an optional deeper parallel track on the heavier days would help us spend more time at the frontier.

Other thoughts3

Drained because of sleep schedule, the day was well organized.
no
Enjoying it

April 2026 Sat 4 Apr Leon Lang 14 responses energy 2.9/5

Called “AI Alignment: an introduction” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.9 of 5

Teaching and format 3.7 of 5

Difficulty match 2.6 of 5

Energy 2.9 of 5

Read what they wrote35

Worked well12

The amount of time per reading was good, and the reading guide in the doc was helpful (as were some of Leon's explanations)
Discussions
I liked that we took the time to do many 1-1s (ideally with everyone, but that unfortunately didn't work) to get to know the other participants.
Guided discussion questions! and freedom of interpretation
The idea of 1-1 and breakout discussion groups.
The individual reading then discussing it amongst ourselves was a good way to learn and learn from what others had understood from the same articles/papers.
Meeting other participants, having discussions about perspectives on AI alignment.
Meeting people in the morning was really good, eventually had issues with not having meetings synchronised, so slightly more structuring might help. Discussions on the reading was interesting
Reading time was nice
Discussions, the people
1:1s were fantastic!
I liked the content! I also liked the discussion in small groups.

What they took away11

I think the general background was useful, thought I wouldn't attribute this to one specific resource
I don’t know the term CEV somehow
Reinforced concepts of alignment as a problem and definitions
Alignment target: there's inherent problems to how we can formulate what we want. Intent alignment: how we can tell Agent to do what we want it to do. Not what the right thing to tell the agent is. But what is it that we want it to do? CEV, alignment targets; alignment problem decomposition: specification gaming. Shard's theory.
The view that AI is acting in a context and that we are actors of this context. Such as when AI agents wait for our prompt before acting.
I learned many new concepts (and names of ideas) in alignment - eg inductive bias. Helps to shape arguments about these things and identify problems to attack.
Refreshing if basic ideas was good for me
There was one article I had not read yet, the one on tool AI vs agentic AI. That was good to read
Maybe reading about process oriented learning or training stories
Singular Learning Theory, also met a person whose interests me greatly.
The paper on how agent AI is more likely, even though I have criticisms of it but this led to interesting discussions.

Would improve12

Not all of the readings had useful information to length ratios (e.g. the inductive bias and rocket alignment posts), but this could be got around with Claude summaries
Grouping people / systems that help everyone contribute in discussions
The 1-1s were a bit chaotic and we didn't have time to do them with every other participant because of that.
More structure in information presentation, more organization, slightly lower quantity or literature or higher density of information/simplicity, more context surrounding literature and non-academic sources/critical evaluation
1. Have a more structured way to allocate participants for group discussion. and yes, having moderator involved in the group works nice! 2. The food today was too spicy. 3. Reduce the number of discussion times? 4 times of reading and group discussion back to back is tiring. And also having consistent discussion group set is better than keep swapping it since we can reference points said earlier and synthesize insights better.
I would give more time to read the articles and also have a whole group discussion after the smaller group discussions.
A lecture summarizing some of the concepts might have been helpful, in addition to individual reading and small group discussion.
Questions on the reading maybe could actually be closer to just suggesting to discuss your thoughts from them, as to me 40 minutes for the readings was not much, so just thinking about them is probably the best option. Increasing the 40 minutes might help, although clearly there’s not much extra capacity in the day schedule.
It feels like it was a bit too basic/too much I have already seen before. Not much novel discussion. But good to get people up to speed. I would have preferred more reading/personal learning time and short general discussions with the full group than these longer discussions with split up groups. For getting to know eachother it would have been nicer to have a round around the table quickly and then just dive deeper immediately. Also maybe just a brief overview/presentation of the material might help the quality of the discussions.
I have ADHD so my ability to read for a long time is very diminished when compared to others. I'd prefer video based content, ideally video recordings of presentations, since I usually need to 2-3x them in speed to be able to focus on them and not be bored by the speed (which is why I have a very hard time listening to real world talks as well).
Seating was a bit uncomfortable. Lunch was too spicy for me (I'm sorry 😭)
Ideally having the instructor assign somewhat different groups for each discussion moment would have been better so we don’t end up discussing with the same people each time.

AIXI 2 runs 48 responses, 66 written

for pasting into Claude

September 2026 Tue 15 Sep David Quarel 36 responses energy 3.6/5

Called “Solomonoff Induction + AIXI” that cohort.

Rated against what each person expected. 0 is as expected.

Overall -0.09 as expected

Content -0.23 as expected

Clarity -0.23 as expected

Motivation -0.43 below expected

Exercises +0.09 as expected

Readings -0.13 as expected

Other activities +0.14 as expected

Pace +0.40 too fast

Difficulty +0.26 too hard

Novelty +1.40 mostly new

Read what they wrote54

Worked well18

I think both the lecturing and the notes were very good.
exercises with a partner really made a difference!
It was easy to ask questions and the answers were helpful, but I don't think that gathering almost all the interesting information in personal communication in such group sessions it good
The lecturer was clear
- David was a really energetic speaker, made a heavy topic easy to stay engaged with - Good worksheets, and the balance of short lectures with lots of time on exercises
I like that it used policy and Bellman equation -- the material from yesterday. Same goes for Thursday-Friday transition with HMMs, and Friday-Monday flown to Tuesday seamlessly. Very nice. Also the nice and long list of exercises to work on by yourself.
I liked that the teacher was very eager to answer questions, I liked the explanation on KL divergence, and him recommending me a concrete section in the book that exactly answered my question
David is clearly excited about and deeply interested in the material; he did a very good job with questions both during and after presentations.
The website content is FANTASTIC, well done.
Personalized explanations.
Long exercise sessions dedicated to learning by doing. We went over all proofs ourselves rather than watching someone zoom through a derivation. Was very nice.
The clear layout of notes and encouragement to work in a way that suited. I also enjoyed listening to the questions asked by other colleagues.
working through the exercises which seemed very well written out
The abundant time for exercises and discussions.
The written lecture and the exercises were great, well written, and built up to interesting result. However, I question the usefulness of relying entirely on exercises to carry the core information. I am not necessarily interested in solving mathematical puzzles, but see how the experts in the field do things (because one day I might collaborate with them). For example, several exercises were either straightforward or tedious according to notational rigour, and I would have preferred seeing the lecturer carry out at least some solutions or proof by hand to see what is the style of the subject.
The website to the lecture materials and the problems was very accessible and the sections were well written.
The exercises were fun but still for someone with a background in bayesian stuff it wasn't very new, but perhaps for others it was useful.
The focus on the exercises + the incentive to work in pairs

Would improve24

Maybe a more balanced distribution of lectures and exercises - while I found the exercises to be interesting and reasonable in difficulty, it was a bit "intellectually exhausting" to work through them for multiple hours. I also would have liked to see more material on the current research paradigm in AIXI and what it hopes to achieve.
Needs a bit more context and applications. It all seems a bit far from what is happening now in the world.
It's surely teaching style, but extremely short lectures, not engaging constructively with questions, and exercise sessions that are mostly symbol juggling together don't show how this all is relevant to safety and alignment.
I think adding more lecture material and not deferring everything to questions and exercises could be much more helpful. If felt like we were given some abstract nonsense introductions and definitions, and had to clarify everything ourselves. The connection to alignment and reasonable agents we observe and want to make predictions about it also kinda vague. If there are known results about AIXI application to known environments (e.g. Markovian), adding them to presentations and allowing to play with them locally could be much more convincing.
I really enjoyed the lectures. I think today would have benefited from more lecture time. The exercises, although nice, were a bit difficult to parse (there was a lot of notation) and probably something that would be better to look at in our own time.
- A bit more intuition before the proofs i.e what the Bayesian mixture is doing on a small example
This day's structure (i.e. very short talk in the morning and afternoon and then exercises) did not work the best for me. The exercises were mostly around proving the results from the talk but most of the times I didn't know if I was proving something that was just a middle step for the big proof a few exercises ahead OR an important result that I should be able to interpret in the big picture of Solomonoff Induction/AIXI. I think a few changes that would help me would be (a bit) longer more comprehensive talk AND/OR more discussion in the notes in-between the exercises. (Please note that I'm quite in favour of the philosophy behind this structure, I just didn't feel it was accomplished in the best of manners)
I feel the model is too far from any computational process that could be implemented in the real world to be any useful
Today was a big jump in terms of usage of technical terms. Complexity, computability, etc. had not been discussed previously and this is a hurdle for those with a weaker background in computer science, including the quiz at the end of the day, likewise (but to a lesser extent) during the lecture. Regarding the lecture, I strongly believe that taking more time and perhaps showing one of the proofs (possibly partially) would help. This could also break up the long exercise session during the afternoon. But generally slowing down the pace of the lecture would be good. I heard from multiple people that they had to go through the lecture material again, because it was too fast in the morning. In my opinion, it is important to answer questions and confusions during the lecture even if that means slightly adjusting the schedule.
sadly for me the lectures were much too fast, felt quite very rushed, there was little space to think about what's on the slides and I left both lectures not knowing what actually just happened :(( I would have benefited from a less rushed presentation, more motivation of what we are doing, more coming back to what all the notation means, and importantly again how this fits into ai safety research. we asked how the content today is relevant to ai safety and David provided a reasonable argument but I really think for such highly theoretical days it's super relevant to embed things into an alignment perspective
The presentation was generally quite good. However, it introduced several ideas at once; I wonder if it would've been clearer to separate them. We could've started by discussing optimal prediction and optimal behavior under an arbitrary prior/hypothesis class. Most of the results we discussed can be two-thirds proven in that setting anyways. The Kolmogorov prior is cool, but I think it makes more sense to introduce it afterwards, since it comes with a bunch of philosophical and technical questions. I felt like these got somewhat mixed up with the relatively straightforward general-optimal-agent story. Also, more discussion of how this relates to AI safety would be good. I'm not sure if this is a complaint I have about today's presentation or about the actual state of the AIXI field; quite possibly the latter. The exercises were reasonable, but not a great notational complexity to conceptual complexity ratio. I ended up spending a lot of the allocated time working on unrelated stuff anyways, so maybe the later exercises were better.
This was a really interesting day. There is a lot of material so I would go back to two days (delete the world models day!) and this would allow you to cover too recent papers. ie. https://djsutherland.ml/papers/nfl-ait.pdf seems quite interesting and show a bit what AIXI labs is doing. From the lecturer, we got the impression that AIXI is "old stuff" that might not be useful, but seeing what these people are working on might give another viewpoint. Overall the format was good, I enjoyed the zap lecture and then doing a lot of self-study. Maybe going together through a paper (e.g. the one I suggested) elicits a deeper understanding of the material + fruitful discussion between people.
Too much independence, and the lectures were so fast. I would have liked more time for teaching and less independence.
After a week of mainly mathsy work I'm really yearning for a link to practical alignment. How could these frameworks apply to analysing transformers? How are these methodologies applied for interpretability? How would the interplay of theory and empirical work look?
a bit rushed at times and sometimes I wanted to hear the question and answer as I would not get to hear it later.
more discussion about the answers as a group :)
A better way for slides to be presented. The TV screen is simply way too small and not conducive at all for a presentation to 40 people.
I would like longer sessions and less time for exercises. Overall I find this approach a bit too formal, but this is personal taste
The room layout and lecture setting was more conducive to coworking than teaching. This was clearly by design, but I found the format offputting for a completely new subject.
I would show more concrete empirical research examples instead of just going through the theory.
In my opinion, today was too much exercise-based. I would prefer to discuss the theory in detail in class, so I can do exercises in my own time later. The first few exercises were just math manipulation which I personally find a bit tedious and not very insightful. I would much rather learn the theory in the context of AI safety, rather than proving theorems.
There were many exercises that were not giving the insight I would have expected and the notation was a bit too heavy. There were to some extend a bit of overlap with yesterday's session, but the theory was still interesting.
Talk was slightly too fast for me but I understand the philosophy of the lecturer. But maybe a tick slower, but then again I was slightly late so its on me.
Maybe adding one or two recent paper to showcase current research in that field

Changed my mind10

I think AIXI seems fairly promising, but I didn't feel like I got much context on what exactly it aims to prove.
not sure
n/a
That even so called weak assumptions might not be that weak. Also, infinite compute, really? I do share the sentiment that we have to be studying idealized agents, but significant amount of time should be devoted to realistic scenario too. And that's where the math has to be employed the most, as all these theorems might well be inapplicable (like no one in the market cares about the existence of Nash equilibrium because no one bothers computing it and you can't really arrive at it in reasonable time).
How useful aixi is
I used to think SI smuggled a very strong assumption into the choice of the Kolmogorov prior. Now I think it smuggles a very strong assumption into the choice of UTM instead. (This is a real improvement in my eyes.)
This feels particularly useful to derive mathematical bounds on agent behaviour- but not a lot of people work on this although it seems quite interesting. I feel like this can be more promising than things like computational mechanics (which feels like fancy mech interp). I think the roadmap on how to use AIXI is somehow fuzzier but more promising.
I discovered AI safety initiatives in Barcelona, my city.
This work is too theoretical so having an impact.
I understand now why AIXIs might be useful to AI alignment, albeit not immediately.

Other thoughts2

n/a
Today I spent most of the day reading Simplex papers because I found that much more interesting than the Solomonoff stuff.

June 2026 Mon 15 Jun David Quarel 12 responses energy 3.8/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.9 of 5

Teaching and format 4.2 of 5

Difficulty match 3.1 of 5

Energy 3.8 of 5

Read what they wrote12

In their own words9

Very good notes and carefully written exercise sheet. It covered some of the gaps (imprecise definitions/ non uniform notation) in the notes and problem set from day 5.
The day was structures well and the handouts were nice. David was very energetic and very helpful, and got me interested in the topic.
the lecture portion was fast, which I really appreciated! it was nice to spend most of the day just working on problems.
the solomonoff to aixi arc was the strongest thread so far: universal prediction, the bayesian mixture, the cumulative bound, and the k(mu) ln 2 complexity bound as a formal occam's razor built cleanly into the universal agent. the worked exercises were the highest value part. the safety caveats, like non exploration, wireheading, and bad priors, made it much more grounded. would improve: a bit more connection between the end of deck agent taxonomy and the alignment motivation, since that is where the clean theory seems to stop.
- I didn't like the focus on exercises - I feel like more discussion time, or a more extended lecture would have contributed to getting more take-aways from the day - In particular, I would have liked the connection with safety to be clearer
I liked both sessions. The two presentations felt a bit rushed, but they were nicely complemented by the worksheets that summarised the main points and definitions. I still don’t see how this formalisation of general intelligence could help us gain more insight on modern architectures and obtain practical results, but I’ll wait until tomorrow’s session that will be more hands-on.
exercises in pairs was fun, the lecture wasn't too long, more time for exercises
I really liked the teaching format, but it was a clear case where it's difficult to relate to the utility without some historical context. More of that would be great. Even if it would be of the sort: "Well we wanted to answer such such and such, but that's actually really hard because so and so". It's a rather abstract topic, and I'm not quite convinced that AI will move naturally to a direction where this is a good model.
Given David's perspective on the usefulness of this content, I wonder if there would have been a more useful topic to cover. I really did enjoy today though!

Other thoughts3

The in-session worked exercises and the live derivations were the best format. More of that.
I am personally considering working on AI safety when I finish Iliad, and I suspect other participants are too. I think it would be nice if every teacher highlighted the most promising research agenda/ research opportunities related to the content of the day. I feel like this has been mentioned on multiple days, but it would have been great to devote more focus and more time to it. e.g. give a 15 min explanation of recent work towards this research agenda; instead mentioning it for just 2 min at the end.
It was super cool to meet such an inspiring and sharp person as David! :)

Alignment in Practice 5 runs 119 responses, 170 written

for pasting into Claude

October 2026 Thu 8 Oct Margot Stakenborg 32 responses energy 3.9/5

Rated against what each person expected. 0 is as expected.

Overall +0.57 better than expected

Content +0.53 better than expected

Clarity +0.57 better than expected

Motivation +0.67 better than expected

Exercises +0.78 better than expected

Readings +0.46 better than expected

Other activities +0.76 better than expected

Pace +0.03 about right

Difficulty -0.17 about right

Novelty +0.50 mostly new

Read what they wrote38

Worked well12

I liked the debate in the end :) Good way to help memorise the various topics.
the structure, well partitioned between lecture, reading, discussion
The instructor-lead parts were focused and clear.
The blue/red team exercise at the end was very helpful to make concrete the techniques we read about
The final game we did with Margot was very nice. It helped me touching by hand the challenges of making a model safe and preventing from attacks.
great resources and amazing, enthusiastic, enthusiastic, knowledgable, clear, and available speaker
The red/blue team exercise was extremely helpful for consolidating concepts introduced today
Margot's delivery, selection of material, and engagement with questions, is excellent. The discussions were helpful. The debate was enlightening, and clarified some of the key points of the day.
The exercise at the end
The exercise at the end
The exercise was a lot of fun.
I liked the activity at the end. It helped me learn at a deeper level.

Would improve15

I found the debate enjoyable and interactive.
Fewer topics but a selection of modern/popular topics in more detail.
It was great a overview, slightly more quantification would have been nice, e.g., real data to and consequently discuss texts like AI 2027 and Europe 2031 from another critical angle
For this track, if the goal is to expose those new to the field to the fields' essentials, I'd cover more material during the instruction phases and leave less for discussions.
The slides could have used more visuals maybe? I think today was very good
the debate was a bit of a miss for me, as no one was cyber-expert so we had no real basis to have an informed discussion / debate. i would have rather spent more time listening to speaker or simply
The discussions are good, but could be a bit heuristic and a more concrete code-based tutorial would be nice to better understand alignment in practice
A bit of overview of the papers/readings before unstructured reading time to make sure we hit on key concepts
nothing
SGTM is awesome! Some more time on it -- the mathematics -- could be helpful.
Nothing
This was a great day: I genuinely don't think there was much to improve. Perhaps during the example of redteaming/blueteaming to not give specific examples and instead let us figure it out on our own.
Maybe spend more time on SGTM and gradient routing.
Lectures were very professional. I would have liked more time for the game at the end, but I don't know if this is even feasible. My low energy on the day had nothing to do with the lectures themselves.
I think the slides could have been more detailed. The reading was a lot more informative than the presentations.

Changed my mind10

I did not realise that most of the effort of technical alignment does not go to pre-training.
The day has spiked my interest in the red-teaming approach to safety.
I didn't
I'm much more aware of how even policy and governance decisions can have a technical and research aspect to them like tradeoffs of pacing AI development and deployment
Jailbreaking turned out to be even easier than I expected.
The safety of open weight models
Safety for open weight models seems almost impossible
Alignment in practice seems like a Sisyphean enterprise, and I think that it would be worthwhile knowing the tradeoff landscape more quantitatively.
The fact that some of these techniques work, tells me a lot about the structure of LLMs that I didnt know before. It really helped to get more perspective/clues on how to think about the model internals.
I learned about personas and AI as a way for aligning

Other thoughts1

Generally, given the mathematical expertise of the audience, erring on presenting mathematically complete material (when appropriate/available), would be welcome and, I think, efficient.

September 2026 Fri 25 Sep Aden Power 35 responses energy 3.5/5

Rated against what each person expected. 0 is as expected.

Overall +1.06 better than expected

Content +1.25 better than expected

Clarity +0.84 better than expected

Motivation +1.44 better than expected

Exercises +0.63 better than expected

Readings +0.14 as expected

Other activities +0.11 as expected

Pace 0.00 about right

Difficulty -0.13 about right

Novelty +0.50 mostly new

Read what they wrote55

Worked well15

Great teaching!
I think breaking up the lectures with the game/reading/etc was great, overall just a great day. Aden is an incredibly entertaining and competent lecturer :)
- Walking through the pipeline phase by phase (pretraining, post-training, deployment) gave a clear structure for where each alignment method fits
Practical overview of post-training affordances and empirical failure modes.
Everything landed; best session of the programme
The lecturer's enthusiasm and a lot of real-life examples + rumours and reports about real companies and state of the world. Aden added a lot to the lecture in this way, by telling about firms etc. Very grounded in reality lecture, I loved it. Resembled Julian days 1-2 in many ways. I also loved the Forbidden Techniques discussions, those were extremely informative. As well as train / val /test sets.
Great energy from the speaker!
it was great
The small game we played was surprisingly instructive. The discussion about the progress of the "practical" alignement field was very important for me to understand the concrete steps being taken at the moment, definitely a core lecture module for the intensive. Kudos to the teacher, who managed to remain engaging and clear through a very long lecture day!
Aden's lecturing was excellent once again. The content was chosen very well too. Very grateful for the effort Aden must have put into the last few days - thanks so much.
The content was amazing. Felt super relevant, it was clear why its important and how its useful. It was also amazing context. Wish more days had similar content.
I think the presentation struck very good balance between technical details and general ideas.
Motivation with examples.
Very relevant content and I can tell the teacher is very passionate + knowledgeable about the literature and developments of the field
The speaker.

Would improve23

I would need more breaks to follow the entire day
Maybe some discussions ran a bit off-topic, but it wasn't too bad at all.
- More time on post-training (RLHF and the methods after it) as a proportion of the day, since that is where most current alignment work happens - Links to the key empirical papers mentioned during the day, to read afterwards
Readings didn't add a ton beyond what was already explained in lecture, or perhaps it wasn't clear what we were supposed to get out of them/which parts to focus on.
Nothing.
More time analyzing concrete lab safety cases.
Nothing comes to mind
It was cool, but it was a bit too much talking in one day. It can be split in two
Add breaks in between 1.5-2h lectures. Also, I'd perhaps decrease the information density delivered vocally. During the lecture today, I was recalling Julian's lecture on Day 1 in MechInterp, which was exemplary in many ways. The slides were very informative and with optimal amount of info -- one figure per paper, few words per slides but many formulae because that's what belongs there. Here, on Day 15, the slides are as many (100+), and they are informative, but we do spend quite a lot of time on each I believe, which destroys pacing. Slides should be changing more often, I think.
Awesome day.
The lectures were really great, but I think a couple of more breaks would have been appreciated.
maybe one more short break after lunch would have been nicer
The lecture was very good and very well motivated. The framing of part of the content was also somewhat "historical" (i.e. following the evolution of the field). I have found that to be very helpful, and this perspective may be worth to stress further through the material. The lecture also has many short examples of techniques being used and the ideas behind them, this can be a but overwhelming at times, but also very interesting. Maybe cutting 1/4 of the examples and expanding the others would be good (the rest can be given as reading material) Overall the lecture was very successful and I would not change much!
I think, just in terms of pacing, it was difficult to maintain focus for multiple lectures of this length, despite the fantastic lecturing and content. It being a Friday may have contributed! - maybe this schedule is better suited to earlier in the week? The game today was maybe a bit too simple to feel meaningful. I struggled to follow the section on GRPO a bit.
Today was the best day so far!! Loved the content. Would've loved for it to be split over two days to maximise retention since the content was so so important.
This day was awesome but could've easily been to days. We had days with a third of the content here.
Do breaks after one hour, no 2 hour or 1.45 hour lectures
I really enjoyed today bcs all topics are super relevant. however for my taste there was too much lecture/ passive engagement and not enough discussion/ active engagement. it might have been nice to distribute this day over 2 or 1.5 days and add more discussions in between.
It was a very interesting day but way too long. We sat for like 6 hours of lectures and however interesting Aden is that is simply too much. The content is very useful for people (although broadly irrelevant if you've done any applied alignment stuff) so maybe it's worth splitting it.
more discussions on whether other types of persona could be useful for us + how we expect continual learning to look like / rough predicted timeline on when it would emerge
I thought today was the best day. Aden was a very good speaker!
Nothing. The day was fascinating.
Lectures were a bit long and I found myself not following at different times. Maybe switch with readings a few times during the day?

Changed my mind13

Learned about some ai safety hackatons at ARC
More differentiated understanding of risks
Slightly increased optimism that, given cleverness and perfect implementation, we can find adequate ad-hoc alignment and control strategies for another year or two of scaling.
Clearer view on how RSP commitments translate to technical evaluations.
Safety cases feel more tractable than I assumed
I think it stayed somewhat similar to BlueDot Technical AI Safety, just way more informed. It does tie well with some of the reasoning I was doing regards train / val / test sets as well, was very useful to hear about it again.
i got convinced that doing safety finetuning directly on internal activations is not a good idea
I think today's lectures made me more pessimistic about the current state of affairs than I was previously - but it is probably net positive that I'm more accurately calibrated now.
Frontier models can get out of control very soon
still not sure how to think about evals
n/a
Control techniques are more diversified than I previously thought
I will not use the forbidden technique

Other thoughts4

Very engaging and practically relevant.
I can't find the MechInterp slides, could you please point me to them? Neither on Discord nor on Github nor on the websites
How are you interpreting these answers? Like, the pace was quite fast for someone who's never seen the material but it's tautologically slow for someone who's already seen everything. How people interpret the questions will vary, so answers should be interpreted conditional on how you think most people will interpret the questions. I think you might get more information overall by having 10mins chat with each of us individually to ask questions. I have a lot of ideas (and concerns!!), and I know the same is true for many other people.
n/a

August 2026 Thu 20 Aug Garrett Baker 25 responses energy 3.7/5

Rated against what each person expected. 0 is as expected.

Overall +0.16 as expected

Content +0.12 as expected

Clarity +0.80 better than expected

Motivation +0.96 better than expected

Exercises 0.00 as expected

Readings -0.04 as expected

Pace -0.40 too slow

Difficulty -0.68 too easy

Novelty -0.16 an even mix

Read what they wrote29

Worked well8

Readings were well chosen.
Having someone yap about a slice of what's going on in empirical AI safety was helpful.
already covered
I liked talking about what frontier labs are currently up to! I think this day should certainly be kept in because it framed things not terribly.
I liked the speakers, I think both had great personalities and passion for the topics they discussed. I appreciated that Aden's enthusiasm did not diminish during the afternoon slump. Possibly my pleasant feelings are because the concepts were easy to understand in English (vs. some complicated notation) and did not take too much cognitive effort to understand.
I wish the first lecture was a bit more as in it felt like just being read off of slides. Second half was awesome
Reading list and discussion was useful. I like the light brush on Borda and a touch on Arrow’s Impossibility Theorem. I wish we go into more details.
The readings & discussion about different approaches to AI control were useful and mostly new to me.

Would improve11

Last night at the evening session, someone made a really interesting point about how toy models should be motivated by experimentally observed phenomena in non-toy models. I wish today had focused more explicitly on those sort of connections between "alignment in practice" and theory, as opposed to just "alignment in practice" in isolation. Especially since control seems like it could have a theory component (e.g., one of the papers talked about how the strong AI was only allowed to send short messages to the weak AI, which seems kind of like interactive proofs from complexity theory).
Problem sets would be useful.
This should've been first or second day.
More perspectives would have been good. The fellow (Aden?)'s perspectives were good to hear but I assume they were more pessimistic than the averags knowledgable person's perspectives.
I guess today's session was envisioned as a bit more chill and not so technical from the start and it was a good interlude between all the maths heavy days. The one thing that worked very well was providing all the background lingo and knowledge AI safety researchers know well and use, and putting the whole development of LLMs in historical context. I particularly enjoying the way it was kicked off, with the presentation about what really goes on in RD at the labs, from the transformer architecture to fine tuning and RLHF, and the various alignment problems and approaches. The whole day was one of my favourites because it provided a broad conceptual picture of what actually is done in the wild.
I feel like this day is a bunch of concepts that people with epsilon familiarity with AI Safety know, but then yeah, if Iliad is getting people that never even did bluedot for example, it makes sense for the day. One way I believe this would be better is if this was in order like in the actual second day, or maybe cut this to half a day. I would've preferred to have one of the cut topics in place of this (like RL - it'd be mostly review for me but at least I review useful stuff - or AIXI - am not an expert and would benefit from learning more). In general, I feel like the order of "A1,A2,..." should be respected.
I think this day should happen possibly on the second day, as I feel like most of the concepts today have secretly been alluded to already/ assumed to be familiar to us and so it helped frame a lot of things from previous days. Today also felt more big picture, so I personally feel like it would have been better to get this larger overview before diving into the more concrete mathematical content e.g. SLT, MechInterp. It's definitely possible that I enjoyed today's material as much as I did because of the background I gained earlier in the week... Also, today's lecture felt more related to Launch day and it would have been better to have more time to stew on this.
I feel like there was some overlap with the first day. Also the llm training stuff was already pretty well known to me. Maybe some of the material can be merged.
would have it earlier in the program since it was more introductory
I’m quite torn because the class materials are important to know about, but the content is boring, as is the problem with the short-term alignment agenda. I taught a similar class before for BlueDot Impact, and I think you could rebuild the class structure and materials in a similar way because they had clearer takeaways about why the material was important. Maybe it is because this is mostly a theory course and I'm a theory audience (so I'm very biased), and most of the other classes are more exciting than this one. I actually think the first alignment class by Leon was pretty awesome and the way the material was structured were useful for a conceptual day, and you could perhaps follow a similar structure. Even with the SLT class, which I didn’t fully follow because it was too fast for me, I still felt like it was more worthwhile than today’s class materials. I think the readings for AI control were useful and well selected. I actually think that the World Models, Computational Mechanics, and Data Attribution days have been among the best-taught classes so far, both in terms of content and format. However, I’m also familiar with the underlying concepts by proxy because I come from a control theory background. They were very clear about what to do while giving you the freedom to learn in whatever way suited you best.
Might be a good idea to move the AI alignment in practice day earlier on? (At this point, much of the content was mentioned or at least hinted at during the other tracks.)

Changed my mind7

I was indifferent to the control literature, but after reading some papers on control I feel I believe control is much more important and worth spending time on.
No
AI Safety is hopeless. I am looking into becoming a monk, and if that fails I will become an electrician.
I realised I should've stopped betting against AI a long time ago and sort of recalibrated my epistemics.
I remembered that empirical work is cool
It made it more clear why safety work is important, how the models are so bad in some respects and the role of the big AI companies in contributing to the lack of safety
I'm wondering if theory is completely not useful at all in practice and that's why people now seem to care about AI for Societal Impact type of agenda.

Other thoughts3

The preliminaries on transformer were super nice! I kinda think this day should be taught earlier in the program
So _this_ is what empirical alignment research means. I retract any previous comments of mine to the effect that Iliad is more empirical than I had expected.
Sorry for only submitting this now...

June 2026 Tue 23 Jun Evžen Wybitul 14 responses energy 3.4/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.2 of 5

Teaching and format 4.0 of 5

Difficulty match 2.4 of 5

Energy 3.4 of 5

Read what they wrote14

In their own words13

I think the debate worked really well.
It was quite nice to have a red blue teaming exercise But I felt like we had too many small discussions for very broad topics Topics were quite nice and good to know and I am also planning to read the additional papers mentioned
walking the whole pipeline (pre training, post training, deployment) and re deriving each sota method by discussion built real intuition for the affordances each phase gives us.
- I loved the blue/red team exercise - I think the content today was very broad, and I really liked that
Too much lecture. could make the same points but with less elaboration. The debate session was fun.
The empirical approach to safety is interesting and important to learn about, since it is currently the only one that can be implemented and tested in more interesting scenarios. The exposition could be more structured. I did appreciate that it is difficult to organize an interconnected series of topics in a sequential way, and the speaker tried, but I felt that it was mostly cosmetic and more iterations of this lecture could probably help him restructure. I would have appreciated more details on the efficacy of the various techniques and a little more specifics. Maybe something more along the lines of an overview of the literature, with a few deep dives on the actual methodology of one or two select papers.
The discussions were well moderated, content super interesting
I think this was all great content for folks in the intensive to be aware of, but I'm not so sure how useful it would end up being for someone doing technical alignment research. At the very least, I've developed more vocabulary for reading/discussing general AI safety papers. More exercises or demos would have been appreciated - I felt like I was sitting for a long time during the last block. The content was all well taught and easy to understand.
I did not like the structure of the day, and the lectures were very long and boring. I think the topic is a very important topic, but the teaching regarding it was not that great. The discussion part was good.
First half of the day was a bit tiresome with the continuous non-stop lectures. Second half was better with the red team-blue team exercise. I feel the materials were interesting but I don't think I have any great takeaways in terms of what I should leave with from the topics that we covered today. Also perhaps it would've been better to have some exercises or tasks where we got our hands dirty with the models, or at least had a walkthrough of a gradient routing implementation or other post/pre-training alignment techniques. Today felt too birds-eye-view/"this is how it works in principle" than it needed to be.
By far, the module on practical alignment and safety that I enjoyed the most so far. I really much liked the exposition and I found the blue/red-team session very instructive. I also appreciated that the speaker introduced themselves at the beginning of the session, and I think that the other speakers/lecturers should also comply with this.
very well presented, very engaging. too much text on slides, sometimes. discussion groups were very nice, the red-blue team game was good. content was a bit boring/slow at times (maybe because I knew most of it). also could have come earlier in the Iliad
The lecture gave a good overview, the discussions were interesting. I liked the red-team blue-team discussions to get an impression on the difficulty of practical AI alignment. Overall, the topic has been noticeably less complex (less formal and mathematical) than other topics.

Other thoughts1

the blue red judges design challenge on bioweapon uplift was the best format, i would welcome more adversarial safety case exercises.

April 2026 Tue 21 Apr Evžen Wybitul 13 responses energy 3.9/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.5 of 5

Teaching and format 4.4 of 5

Difficulty match 2.8 of 5

Energy 3.9 of 5

Read what they wrote34

Worked well13

Excellent format, teacher, and content today! Everything was great, and I especially enjoyed the red team/blue team activity!!
Group discussions and especially the debate - with genuine pressure to think about problems and participate (because answers would be shared with the class)
Evzen is just really good at presenting and guiding people. The discussions were super good, we covered so much ground. It was very energizing and just a very useful day overall.
Debate format I found so much fun but he is such a good lecturer that maybe it’s better to just lecture per unit time I’m not sure. Regardless the debate was so much fun I feel like it improved overall experience of Iliad for some people including me
The content was very interesting and with examples on SOTA models and real world implications. The debate format and group discussions kept it very engaging.
Great format, great presenter (with a good balance between letting us discuss and bringing us back on topic/schedule), very relevant content. I especially enjoyed the Red-teaming exercise (debate).
It was easy to follow, much gentler and more forgiving than the more maths-intensive days
Teaching format and regular discussion + debate game were really nice
The pacing of breakout discussions, as well as the content helped keep up engagement and thinking, respectively, this made today stand out very much as a positive learning experience.
Quick discussions & brainstorming, review of paper results, relating discussion to practice, debate exercise worked very well
Lots of papers and examples.
Very useful to get this overview of what is actually being done in practice, and the difficulties of the related problems.
The red-teaming exercise and the discussion sessions interspersed with the lectures were fun, and were good for preserving focus.

What they took away12

Learned lots of different jailbreaking techniques that were super interesting!
Strategies for safety and jailbreaking… as with many days, the material has been largely new to me, and the overview today was very high quality. From a research perspective, today set out the ultimate practical goals that I feel I need to keep in mind when investigating more mathematical aspects.
It was good to get a complete overview of alignment practices! And to try ourselves to build a safety case!
There was a lot of bits of knowledge that were very current about things like auditing success rates that I thought were extremely valuable and useful for me to decide what is important for me to work on
That post-training is where the persona building happens for LLMs but world understanding is mostly in pre-training.
1. The general structure/pipeline of alignment in practice. 2. The general takeaway that SOTA alignment methods are not that complicated (and not that robust).
I am converging on a theory that this programme may have been designed by and for extroverts with ADHD, and the main benefit for me on many days will be the beautifully calibrated reading lists provided by each instructor, which I can take home and do at my own pace.
Intuitions for the whole process, more feeling of the 'spirit' of trying to red-team
There is much in the material to be let sunk in, and many things to remember, which will require an eventual revisit. Latent around it was a new recognition (to me at least) that there are numerous known threat cases being studied proactively by labs, and they take their discovery seriously at least at the level of allocating talent and resources to it (of course the amount of information and interventions still seems alarmingly low but the neglectedness is likely not as bad as, say, 3 yrs ago).
Alignment in practice is hard but takes many different forms with trade off, understanding of literature relating to jailbreak and vulnerability at different stages of training
Persona characterisation of models
Gradient routing seems really interesting a tractable compromise on getting a more explainable model without sacrificing performance too much.

Would improve8

Bit too much material for one day.
Honestly it is just super good, I think more people should do this setup with Evzen it is just really informative.
The conversations could have been slightly longer to let people have more in-depth conversations.
I did not feel like we had enough time or exposure to develop sincere opinions for the debate or the small discussion groups. They felt fake and forced.
I think we could've gotten more out of the debate if beforehand there was an opportunity to summarize and structure the day's learning, to remember and be inspired by it, and possibly it wouldn't have hurt if some bits of the material were skipped because of that. To be honest, this section of the material could do a good job at introducing currently relevant threats, possibly better so than the actual introductory section. This applies mostly to LLM-related AIS of course, and I view it a valid unanswered question how general this can be made vs how LLM-specific it is -- I think there's something to be learned here about aligning various different, narrow purpose models, which we gestured at with biosec risk, but still remained anchored at experience with Claude...
Another day on it with more literature/practical exercises/tests
I do not think the pink sticky notes on the board were useful. I did not hate them, I just think they were pointless.
Perhaps the constraints on the debate could have been clearer, like for blue team “prioritise defining uplift to bioweapons clearly”, and for judges “only one technique per training section”.

Other thoughts1

The lecturer mentioned that the vulnerabilities found by Mythos could also be found by weaker models if provided a good harness. I believe this was in reference to this article: https://aisle.com/blog/ai-cybersecurity-after-mythos-the-jagged-frontier This conclusion is somewhat misleading, as the "harness" involved only showing the models the vulnerable code. > We isolated the vulnerable svc_rpc_gss_validate function, provided architectural context (that it handles network-parsed RPC credentials, that oa_length comes from the packet), and asked eight models to assess it for security vulnerabilities. I.e., they didn't provide the entire codebase. The experimenters used their knowledge about where to look for the vulnerability, and provided this advantage to the models.

Alignment in Practice 2 1 run 13 responses, 12 written

for pasting into Claude

June 2026 Wed 17 Jun Garrett Baker 13 responses energy 3.2/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.2 of 5

Teaching and format 3.2 of 5

Difficulty match 2.5 of 5

Energy 3.2 of 5

Read what they wrote12

In their own words11

This day would benefit from having good summarizations of papers provided to emphasize the group exercises and balance the individual reading time. I am not very sold on the control agenda, but I guess that now I’m possibly better equipped to argue with objections to my style of AI safety research better.
- I felt a bit stressed with so much reading in such a short time, I felt like I couldn't keep up. - I think the builder/breaker exercise was great and very much prompted me to think critically about the alternative experiment - I think the first reading (blogpost) in session 1 was less important. Maybe it could have been replaced with reading a short summary (the ideas were mostly present in the lecture anyway). This would give us more time for reading the paper and breaking/building.
I rated today low on some points because I personally didn't find it engaging and it also was not very convinced that control is more impactful than interpretability. If I had to keep the general topic today, I'd hope to remove all the discussion and overview and just look at specific papers and experiments. The first post we read from in the morning felt by far the most substantive to me.
Garret seems broadly informed, but wasn't able to be too specific about questions. Furthermore, the slides we're aesthetically very nice, but it was also clearly clauded to a point where one shouldn't actually read the slides. This made it a bit taxing to follow along.
the strongest thread was the measurement side: model incrimination, prompt counterfactuals, and decorrelated checks. the scheming vs confusion frame gave me a practical checklist. builder breaker was especially useful because the persona vs goal question is still open. on control, the safety usefulness tradeoff, why trusted editing can beat trusted monitoring, and control as correction rather than full alignment were the clearest ideas. best moment: whether publishing the best red team attack teaches future models to evade it. to improve: discussion groups got noisy without facilitation. a simple template per group, claim, flaw, alternative hypothesis, would help compare papers. track 2 game theory would benefit from one fully worked execute, replace, audit example on the board.
The lectures were sort of boring, I really liked the readings and discussions thing. The idea was more like explore little, but make sure it makes a lot of sense, so the teaching was nice there. Time was given when people needed it and were slow reading the material.
Garrett did well with making the material more engaging but it was (perhaps by design) a bit too reading heavy and hands-off. I also don't know if we are going to use AI control directly for any other topic so unless the plan is for Iliad to also launch an AI control research team, I'm not sure how today fits in with the theme of the Intensive. This stuff useful to know (and some of us have seen this material in other places like BlueDot courses) but not sure if this is the most valuable day for Iliad Intensives.
I think these are key topics to cover. But the slides need work, both in terms of overall structure and in terms of specific content (nonsensical lines). I got almost nothing in terms of overarching conceptual structure, but the "experiments" I read about where interesting.
The contents of today's module provided a complementary view to the theory-based approach we've reviewed so far, and this is good. It focused more on prosaic alignment, namely on the role of empirical experiments to investigate the behaviour of LLMs in various test environments. However, I didn't like how the contents were taught. The reading material was too vast for the limited time we had at our disposal, and I had to rush through the papers. As a result, I couldn't really get all the main points. I think this was common amongst participants, because the discussion was not as constructive as it could have been. I would personally prefer fewer readings, allowing more time for discussion. Finally, I feel that the last presentation didn't actually provide a critique of the approach.
I liked the presentation and format of the reading and discussion groups. I found the google docs too full and wordy to focus properly on what's the current task (some slop in there, e.g. focus on this and this section when reading..→ maybe rather just say once "skim and extract what is useful for the downstream task". The presentation had also some slop in it. But overall, the design was very beautiful and I appreciate that a lot and the presentation was great!
AI slop on slides is not something we should be accepting. While the plan for the day was good, the quality of lectures lower than average so far and again with Garrett, I dislike his order-giving style.

Other thoughts1

i have hands on background here: i built a controlarena monitoring setting at an apart x redwood hackathon and have published work on emergent misalignment, so i would value being pushed toward the harder open parts rather than the introductory framing.

Ambitious Mechanistic Interpretability 1 run 25 responses, 21 written

for pasting into Claude

October 2026 Thu 29 Oct Eric Michaud 25 responses energy 4.1/5

Called “SAEs and Activation Geometry” that cohort.

Rated against what each person expected. 0 is as expected.

Overall +0.63 better than expected

Content +0.54 better than expected

Clarity +0.46 better than expected

Motivation +0.54 better than expected

Exercises 0.00 as expected

Readings -0.17 as expected

Other activities +0.13 as expected

Pace -0.04 about right

Difficulty 0.00 about right

Novelty +0.38 mostly new

Read what they wrote21

Worked well7

Eric's presentation was very well prepared.
The guest lecturers
interesting topics and well explained
I had a lot of common interest and similar research taste as the teachers, I enjoyed a lot the lessons.
The supplemental readings were a nice touch
The slides had lot of concrete examples of what was being described, something that was missing on the other days.
Excellent speakers.

Would improve8

Both lectures were well structures and well presented. I personally liked both sessions because several emperical results were discussed. For future iliad I would dedicate one day for each lecture with more exercises to practive. This might make physics people more happy :)
Some exercise sessions would have helped.
Maybe some more exercises
very good day nothing
nothing to improve, all very good !
I would have included this module much earlier in the course as it provided some insight into how to explore activations in a structured manner. It was only implied/inferred from the material on other days.
Nothing to improve today. The presentation was great -- lots of nice pictures and well-explained what the concepts we were discussing exactly were!
Great talks! Not that much interaction, though. I feel like having a moment to discuss some topics with a neighbour, or a moment to do a quick quiz or something like that, would be a good activating intermediate activity.

Changed my mind6

Sparse auto encoders seem less useful than I expected.
nothing really just learned a bunch of stuff
I migth have moved the work today to earlier in the month, so that it was shortly after covering SAEs
Not much change in opinion, had already seen much of the material before
I didn't particularly change my mind. I was already interested in the possibility of concepts being represented on submanifolds of the activation space, but my interest was reaffirmed and heightened after seeing this presentation.
I did not realise some people consider today's topic as the most (only?) important set of problems. I updated my priors accordingly.

Computational Mechanics 5 runs 130 responses, 190 written

for pasting into Claude

October 2026 Mon 19 Oct Paul Riechers 34 responses energy 4.0/5

Rated against what each person expected. 0 is as expected.

Overall +0.65 better than expected

Content +0.81 better than expected

Clarity +0.42 better than expected

Motivation +0.45 better than expected

Exercises +0.67 better than expected

Readings +0.33 better than expected

Other activities +0.41 better than expected

Pace +0.06 about right

Difficulty +0.06 about right

Novelty +0.94 mostly new

Read what they wrote38

Worked well12

I really liked the exercises. They helped provide intuition for both the morning's lecture and foundation for applications in the afternoon.
I think the exercise of constructing the belief states was useful.
this was an amazing lecture! i particularly enjoyed how Paul spent all morning on the basic notions needed to understand "the meat" ie his actual research work. i think this would be something that increases the effectiveness of most days: spend the morning on very basics anyone can follow, and then progressively move into more superficial presentations of more advanced results
the speaker was great at answering questions
Since the papers by simplex AI are popular in AI Safety circles, I’ve come across them before but to have Paul explain give a whole day of talks and exercises to understand the core ideas has been so enjoyable! I have a much better grasp of the idea now than I did skimming the papers earlier. As always I enjoyed the exercises and learned the concepts better through them.
Content was very interesting
exercises
The exercise was certainly necessary.
amazing discussions within participants and a lecturer. Great environment to develop research ideas.
Paul, the lecturer was very nice and explained things well and interested in talking to us.
I really enjoy the topic, explanations and discussion with the mentors helped a lot.
Paul knew the concepts really well and the presentation was well designed.

Would improve18

Show slides 15s longer so we can read and digest the equations. Everything else was fantastic!
The lecture was really interesting, and I think this is a foundational topic that should be included in any introduction to the field. I feel like I now understand belief geometries well. I did find the tensor decomposition for the interacting agents a little confusing to follow.
one small thing that could improve the experience for the lecturers would be to give them a (possibly ai-generated) summary of the kind of background people have. a few of the speakers expressed uncertainty abt this
Paul Riechers talk was great, but would have benefited from more mathematical precision. A number of statements and constructions seemed to crucially depend on certain assumptions and these were not clear enough at times.
This isn’t quite something to improve but more of a suggestion, simplex AIs papers are somewhat known within AI Safety circles and it was so good to have them explained in detail today. Maybe big results/papers in AI Safety, as they come, could be a good lesson to dedicate an Iliad teaching day to!
Not much, exercises were very good as was the pacing
Have a little 5-minute break every 30 minutes to discuss a particular topic with our neighbour (for example very simply "explain what was just said"), which would help to digest the interesting (mostly new) information.
nothing
Not much, I really enjoy this subject and the slides have so much valuable information.
Would have appreciated more of a look at the mechanical aspects of the theory aspects of newer research papers in the area. Perhaps could have benefited from more structure.
I thought today was very good, but quite difficult. Perhaps this topic from Simplex's research could be spread out over 2 days instead of 1.
I only attended the morning session then it was lunch, then exercises until 2:15 which I didn't attend the evening session. But I don't think we covered experimental applications/findings of belief geometry to LLMs and the remainder of the slides until after 2:15 which constitutes like >60% of the material. I might've preferred a speed-run of the material upfront, and get to the upshots before lunch. Deriving the MSP and belief geometry for some processes is important but I feel like the main idea was already conveyed via white boarding in the morning.
Some more exercises would help. We only had one detailed exercise session today.
I would like to have slides accessible at the start of the day
Overall the lectures are nice. Would appreciate clearer definitions and examples of certain terminologies such as belief geometry and mixed state presentations.
Great lectures, nothing to change here. The material is a little sparse, it would help to have more information in the slides (links to the papers mentioned, etc)
Julian knows his stuff, but for first time consumers of the topics like me he assumes too much knowledge and underestimates slide-fatigue. I would encourage him, to observe how, e.g., Tudor lectured. The pace was slower, the audience was guided through each slide, all notation was explained, all graph axes were labelled, the context was repeatedly given, it was emphasized when things were important to retain and which were excursions etc.
Nothing, great lecturer today + good exercises.

Changed my mind7

I am more excited to dig into belief state geometry as a potential research project
This convinced me that belief states are the right way to think about transformers.
Computational mechanics seems like a really appropriate framework for looking at LLMs. The most convincing one I have seen so far
I did not know that HMMs were used in this field.
Using large language models to predict hidden Markov models produces accurate fractal probability simplicies. Transformers have an inductive bias to factoring into low dimensional components of the activations weights. Factors representing abstract features are generally found in early layers and the syntactical factors generally occur later.
I was not familiar with this direction of research and it seems really promising.
No. I have been interested in computational mechanics for a while so at least the conclusions were not new to me.

Other thoughts1

2nd submission for this day by the way (I recalled some more feedback on my end)

September 2026 Thu 10 Sep Xavier Poncini 41 responses energy 4.0/5

Rated against what each person expected. 0 is as expected.

Overall +0.92 better than expected

Content +0.67 better than expected

Clarity +0.51 better than expected

Motivation +0.10 as expected

Exercises +0.51 better than expected

Readings +0.63 better than expected

Other activities +0.25 better than expected

Pace +0.03 about right

Difficulty +0.03 about right

Novelty +1.05 mostly new

Read what they wrote65

Worked well23

I really enjoyed the pacing and intuition / explanations of the lecturer
Described above.
The progressivity of the concepts presented, also the article with a short summary and a guide to point out the most important sections to read.
Slides were very clear, and the exercises fun and instructive.
Good explanation of generative models, linear embeddings, sufficient stats; good discussions.
Learning new ideas
It was great.
Long exercise sessions, with exercises not being too hard, so it was possible to get a grasp of all the concepts covered
really enjoyed the python exercises for computing belief states and the quiz (good check). also easy to follow "story" of going from hmms to belief states to the transformer result
Clear build-up of definition.
The answers given by the teacher and conversations with him and other participants
Lectures were well structured. I could clearly follow how the ideas were building up. The readings were relevant, and I was impressed I could understand a technical paper on a completely new topic after only a few hours of lectures.
Exercises
Xavier was the best lecturer so far. Found him very easy to follow.
The clear structure with the timetable, it was paced very well with plenty of time for questions. The exercises were at a good level and very clearly linked to the slides and the paper we read at the end.
The structure announced in advance: a 30 min intro lecture with two follow-up deep-dives. Having an intro lecture really helps. The slide numbering -- makes it easy to do notes and refer later with the questions. The exercises -- really drove the point home and assisted with understanding a lot, as it felt a bit abstract at times. Finishing with reading and discussing a paper was a top notch, just like yesterday with Tudor on Physics of Deep Learning. I love this structure so far.
the exercises are very well chosen. The presenter was very good.
I liked how the day was broken up into lectures and exercises/reading
As opposed the past three days, the material, while advanced, seemed more targeted and far more clearly presented in the slides and the problem sheets. The problem sheets themselves were more helpful than those of the past three days as they were detail-oriented and low-level, catered towards enhancing our understanding of the very basics of the new techniques introduced today. I did not feel overwhelmed by them and came away feeling like I measurably learnt something new. The larger amount of time allocated to doing them was also very welcome and made the day feel far more productive than previous ones.
Xavier did a really great job of explaining mathematically challenging concepts clearly in limited time, and also in giving broader motivations for the field.
the exercises were awesome. They really helped me understand the material.
The day was very well structured and it was clear what we were doing when. The exercises were clear and helpful.
Both the presentation structure and lecturing by Xavier were great.

Would improve24

This day was really really good. The pace was fantastic- you could follow everything whilst being slightly challenged. The exercises were good too, and at the end of the session we were able to fully understand a paper (not just superficially) and be able to discuss its merits and limitations.
Maybe adding some animations and some proofs for the most important results in the appendix
Perhaps one example could have been presented inside the joint class. The lecture was clear! Perhaps it would be possible to compound the python exercises to try and reproduce the findings of https://arxiv.org/pdf/2405.15943 in a small transformer. More comments about open problem in the field would also have been useful to give more direction! Overall very good lecture and the topic fits well in the allocated time.
We spent a lot of time introducing and defining GHMMs, minimal sufficient stats etc. This material was very clear and pedagogical, but it might have been nice to spend more time talking about how this relates to learning, how it cashes out in LLMs, and what the important open questions are. I now know a lot about the data-generating process, but still have only a fuzzy understanding of why or when we should expect it to be embedded nicely in transformer activations and what implications this has for behavior and learnability.
I thought the teaching was very good, in fact I wanted more of it! Felt like there was a lot of time towards the end of the day for reading that could have been filled with an additional lecture / some other activity.
The exercises 1 and 2 were really easy. But maybe that's actually a good thing
- more time on the mixed state presentation, it went quite fast relative to how central it is - link the exercises more directly to the Shai et al. paper i.e show how the belief states from the HMM exercises end up in the residual stream
I think at points the material was quite abstract and I didn't really connect it to the feature analysis from say mech interp. Maybe also the applications to steering and how these tools can be applied felt fuzzy.
very nice lectures and exercises, maybe make more time for coding?
More time for exercises, as they were fun and useful.
Commenting more on the limitations and future goals
I would add more coding exercises and show resulting plots as outputs and explain concepts by interpreting them.
The lectures were overall clear. I think it would be beneficial to go through exercise and examples in the classroom.
Nothing really good day
a short explanation of IID, clear definition of hidden markov model at the start.
Hard to say honestly, such a great lecture, structure, the material and the day. Three things though: 1. Exercises -- maybe it's just me, but they required a lot of pretty simple compute, which becomes tedious at some point because you do not feel like you are learning or understanding smth new with the new exercise. Maybe makes sense to have a pointer like "if you do not have time to do the computations, take the resulting matrices from Solutions and proceed to the next steps." 2. Given that I have been catching (although maybe it's just me being unsure about the hidden state of mind of surrounding students) the "we can find the belief state geometry, so what?" vibe slightly, maybe makes sense to provide an empowering example, however small. The paper was incredible, don't get me wrong, but it does not look like it was found too promising of the research direction that necessitated the entire startup established under it. Smth to think about rather than just a suggestion, as that's the problem that was present in "physics of Deep Learning" as well -- people are like "So what?" forgetting that Quantum Mechanics and GTR was also pretty much "We know the deflection of light by Sun and additional 5% of precession of Mercury's orbit, so what?" and then we had precise GPS and gravitational waves. Nice to be showing that "So what?" is not a question to be asking. 3. it would be nice to make the intro lecture somewhat more connected to reality, as it feels quite abstract in the beginning. Say compare this with the hidden states of mind: we do not know the mood of the person, and we infer it from what they say. So we have this "mundane" problem familiar to pretty much everyone and then put the math behind it. Otherwise the beginning might read too abstract, and my brain gets lost when it does.
- Don't assume we've gone through the full exercise sheet when starting off the next lecture (he made a lot of references "as we saw in the exercises", but a lot of us didn't get this far, so it felt really out of context). - Spend 5-10 mins in the beginning reviewing expected knowledge (e.g. HMM for today, "beliefs", "belief system")
second part was a little too dense
I thought the material was very interesting, but I think I'm still missing a bit of information about how this could be applied / used for AI safety/alignment work.
I think this topic could do with two days instead of one.
Maybe a bit more on the foundational math.
- Adding more pictures, - introducing the definitions a bit gentler, I felt that other participants might not have enough relevant background to quickly grasp what's happening (i.e. physicist don't know Chomsky hierarchy)
The lectures we too fast. Make significant pauses before going from one slide to another. It gives time to take notes, digest the material and come up with questions about things that are misunderstood. The language in the exercises was hard to understand coming from a physics background.
I think perhaps the exercises. I found them very easy and often not very illuminating. I also struggled a bit to understand the applicability of computational mechanics to AI safety from the slides, but this became much clearer after a discussion with Xavier and reading the paper. Probably the day could have had more content - I think it is an interesting subfield!

Changed my mind14

I was not aware of computational mechanics and where it fits into the interpretability/alignment world - I would consider this as a potential area of research I would like to eventually contribute to.
I am more aware of the utility to study toy models to assess the learning capabilities of large models.
Somewhat more persuaded that this geometry of learned representations stuff is worth pursuing, although still seems unclear if it will extend to representations of the concepts we care about in the networks we care about.
Very interested to learn about belief state geometry and would be interested to think about theoretical guarantees in this area.
- interp on toy processes with known "ground truth" seems a lot more of an approachable problem than I thought, ended up picking this area for my proposal
I understand better the connection between computational mechanics and theory of world models.
Discovered a really interesting project at simplex.pub, and got interested on computational mechanics
I think computational mechanics is a very interesting field that I didn't know about until today. I will definitely dig deeper.
The reading felt a bit 'numerological' or like astrology but maybe it's just that I didn't understand it well enough. Will think about this.
I am sceptical about this being able to help AI Safety in the next 5 years but appreciate its value in scientific understanding for the future.
Hidden states wow. I knew that HMMs were used for financial markets in the second half of XX century and Renaissance Tech made a lot of money from that, and it's wonderful to see that they apply here too. I also realized that the states of mind reflect very well upon the material. We can be listening to the person / reading the text and be unsure whether they are passive aggressive or sarcastic or normal / whatever.
"pretrained models should learn more than the hidden structure of the data generating process—they must also learn how to update their beliefs about the hidden state of the world as they synchronize to it in context" that's crazy
It was interesting to learn about computational mechanics, although from the readings, it seems unclear how scalable it is.
I have been encouraged to think more about computational mechanics. When I first started thinking about understanding LLMs my first thoughts were a (very) naive version of this approach. I am considering looking into this area further and possibly pursuing research in this direction.

Other thoughts4

Thanks to Xavier for his time and cheerfulness!
n/a
I'm having some personal problems that were making it difficult to focus. But the lectures today seemed great - pace, difficulty, teaching, exercises, everything was really good.
N/A

August 2026 Wed 12 Aug Charles Renshaw-Whitman 30 responses energy 3.9/5

Rated against what each person expected. 0 is as expected.

Overall +0.27 better than expected

Content +0.27 better than expected

Clarity -0.13 as expected

Motivation -0.03 as expected

Exercises +0.17 as expected

Readings -0.20 as expected

Other activities +0.11 as expected

Pace -0.03 about right

Difficulty +0.37 too hard

Novelty +1.00 mostly new

Read what they wrote41

Worked well11

Most things, the anarchistic format, the short lectures, the announcement.
How this fits into the broader perspective of instrumental convergence of goals and how this can act as good deferral
Exercises were great
The exercises.
Liked speaking with the instructors who had quite a lot of good intuition!
- the lecture itself - alternating often between lectures and exercises
People didn't do dumb questions out of context or only for "curiosity" mostly, amazing. Actually maybe today I was the one doing this.
Talking to Ben & Charles was great, very knowledgeable guys
interesting and relevant topics I wouldn't have thought to include myself
Exercises were very instructive once understood.
I liked the excercise which was a guided discovery of the simplex results

Would improve16

It would be nice to connect to control theory and predictive control and tell how they are similar and differences to comp mechanic ala "many faces of RL" in David Silver lecture from 2015. Today lecture and format was awesome. And I like the announcement in the morning to know what to expect. The explanations were also useful.
Specifically today I think everyone would have benefitted from slightly more lecturing time. Even though the notebooks were well designed it was kind of unclear why we care at some points and notation use was sloppy all over the place.
It seems very instrumental to the alignment problem, but the material seems very dense and difficult given that it is fairly new
I think the exercises could have been better designed (I found it a bit difficult to parse it in the time allotted)
comments on "big picture", what is the field currently doing and how it connects to safety
I am not sure what to improve despite thinking it can be improved. I think the instructors did a great job and answered/ motivated the material well.
I think the day could have a lot more exposition in the notebooks on Markov chains and the transformer section; the exercises were honestly not very difficult and the most confusing part was understanding the notation definitions which were not very clear. I essentially only was able to complete them because I remembered the topics pretty well but had I not it'd have been pretty difficult to parse what the questions were asking.
very subjective, but maybe spending a bit more time on the big picture?
As I said in yesterday's feedback form: _please_ tell us in advance what background material will be relevant for a given lecture, so we can learn it in advance. I had a hard time answering the above multiple choice questions, since for all I know, today's content was actually perfectly calibrated to people who already had the requisite background. I'm already familiar with Markov chains, so the morning content was straightforward for me (too easy in fact), but then I got completely lost as soon as it assumed deep learning background.
(small) Suggestion : Turn the notebooks 1 and 3 into a set of lecture notes like the SLT notes written by Kai, improving on the theory, constructing all the details as exercises, this would be VERY awesome (but this is a small suggestion, I generally think it's fine as it is)
Make it very clear at the start how this connects to LLMs and AI safety, and what the goal of this technique is. I think the measure theory part made people more confused on things that were already presented.
I think the problem with the exercises was not so much that they were difficult, but that there was that there was *a lot* of notation being introduced. Also, I often didn't really know what an exercise was asking of me - so I think there is a lot of potential to improve the exercises to make it easier what is being asked
exercise math notation could be improved, make notebook 2 more accessible if didn't fully finish notebook 1
HMMs should have been an initial reading and I would have started faster with the deep learning experiments etc.
Connection to applications. More carefully written exercise notebooks - these were generated by Claude (which is fine), with a typical habit of LLMs without prompting to use jargon and vibe-y language that made it quite harder to follow. Clearer explanation of Hankel matrices: turning finite-rank Hankel matrices into GHMMs, and how this works when it is just approximately finite-rank. This could have been better as a lecture/talk/presentation with exercises to reinforce, rather than just exercises.
I reckon spending a tiny bit longer on the lectures rather than the code would have been helpful but only a tiny bit.

Changed my mind11

I didn't know that comp mechanic is basically predictive control!
This does seem progressing
I did not know much about computational mechanics and now it is my favourite research direction
This belief state geometry approach seems promising
Belief state geometry was more interesting than I expected
leaning further toward empirical research honestly/learning more concrete physics theory
I still believe comp mech is a cool theory, I'm still not super sure Simplex work applies for more than toy models and I still believe comp mech for world models is an awesome direction (that I'd be happy to pursuit XD)
I have a bit more faith in Simplex's agenda than before even though I was already pretty pro-Simplex before
yes world models and yanns focus on them make much more sense now
Added support for understanding ML algorithms as an approach
I had quite a few ideas on possible follow up experiments using computational mechanics that I want to try out so I genuinely left feeling inspired by it. Also I am going to look into Hankel matrices at somepoint.

Other thoughts3

I feel like some mathematicians in the room are too focus on trivial details that aren't that important to the main point like the negative infinity question was borderline annoying.
Context on the feedback: I read "Computational mechanics: pattern and prediction" and "Computational mechanics of input output..." papers from Crutchfield before, I had some context on the Simplex agenda as well (from "transformers represent belief state geometry..." on lesswrong, so of course I know how this fits, but I think people seemed confused about how this relates to safety. Even having read the papers, this was helpful for me because I was always confused on the "bayesian belief update and dynamics happening at the same time" thing and doing the math by hand on the notebooks was super awesome. I also had not much understanding on the GHMM stuff (people be like "negative probabilities how awesome"), so this was helpful. Also notebook 3 was kind of super awesome (approximating epsilon machines? the missing piece from crutchfield papers that I read).
N/A

June 2026 Mon 22 Jun Xavier Poncini 12 responses energy 3.8/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.8 of 5

Teaching and format 4.5 of 5

Difficulty match 2.8 of 5

Energy 3.8 of 5

Read what they wrote11

In their own words10

I am very familiar with the field of comp mech, so this was not new to me. Nevertheless, the lecture was good, the exercises were quite simple even if you are new to the subject. I enjoyed the time in the end trying to build own processes as this tinkering helped understand the relations in intuitive terms.
content was very fun!! the paper we read in the afternoon was a super cool result. I wouldn't really change anything about today
the arc from hmms to belief states to the mixed state presentation was excellent, ending in transformers encoding belief geometry in the residual stream.
I liked the (handwritten) slides. The teaching was good. The exercises were a bit tightly timed, but well selected (maybe with these notebooks it's good to specify precisely what is needed, like "needs an assert". The hints gave away too much, I'd say, but overall I liked the exercises.
I've never heard about computational mechanics before, and I found it nice to have a good introduction to topic. The last exercise, where we had to come up with our process, was fun and instructive.
Content was well-paced with exercises and just incredibly cool and surprising. The last segment was a great opportunity for playfully handling a bunch of the day's material.
- The "Design your own process" exercise was fun and interesting - The coding exercises seemed pointless to me, we were only implementing certain equations, but this didn't really advance our understanding - The explanation of reduction from one HMM to another HMM was very unclear - The notation in the slides was sometimes unclear - I would replace the coding exercises with more conceptual exercises, or at least one of them
The lecture was very nice, the instructor was very energetic and knowledgeable, the exercises and discussions were well designed, and overall it was a pretty good day.
I liked working on coding examples I also enjoyed the topic it was quite interesting
Great exposition and great format, I would have just liked a little more time for the readings, but you can't put everything in 6.5 hours

Other thoughts1

the design your own process tutorial with both geometries visualised was the best format, i would welcome more of it.

April 2026 Fri 17 Apr Xavier Poncini 13 responses energy 3.8/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.9 of 5

Teaching and format 4.2 of 5

Difficulty match 3.0 of 5

Energy 3.8 of 5

Read what they wrote35

Worked well13

Well organized mini-sections with realistic scope, exercises and breaks
Very good presentation and super interesting topic
Really nice hand-written notes and instructor was attentive.
The coding exercises were nice.
The activities were varied but well enough scoped such that neither low level technical exercises nor high level theoretical excursions felt disconnected of the core topic.
The detailed schedule provided impetus and kept us on track. I also liked the diversity of the modules (lectures, exercises, reading, etc.).
Lecturing quality was very good and fluent
The fractal visualisations were obviously beautiful! I liked having a full 30 mins for break in the afternoon.
Great mix of theory, exercises, and discussions, focused on developing intuition about a small example.
The morning with slides and then the afternoon just for readings and exercises worked really well.
Good balance between lectures and exercises and papers
Atomic and very mathematically interesting lectures, frequent exercises, nice paper for reading at the end, being able to speak to the Simplex founders
Xavier did a great job + the lecture slides and exercises were all very thoughtful

What they took away13

Definitions of ghmm's and hmm's, and how nns can internalize them
The difference between belief states and next-token predictions seems super relevant
What is an optimal predictor in comp-mech? How do optimal predictor represent their internal representations? How does this matter to MechInterp? Potentially relating this to my research on "understanding-based eval"
HMM embeddings also show up in transformers.
Bayesian updating in NNs has been framed in the course so far as a compelling but not well supported hypothesis, but since it's implicit in belief state update, the latter's presence in transformers seems indirect but strong evidence that it emerges.
I now understand the Simplex logo :)
Q and A with simplex founders
The intro placing instrumental goals in the context of AI safety and AI progress, with the scale-free concept, was memorable and may change the way I think about other bodies of work besides this one.
Another structured, mathematical way to think about how reasoning may be encoded in neural networks. Good hypotheses to explore.
Understanding how quickly a low number of interactions can become messy and difficult to interpret.
The papers “transformers represent belief state geometry in their residual stream” and “neural networks leverage nominally quantum and post-quantum representations” were extremely insightful and useful.
Found the paper interesting, understanding the HMM theory
The GHMM and "quantum" representation stuff I wasn't very familiar with, so it was useful to have a block of time to go through that. Not covered in detail, but Probabilistic context free grammar were mentioned in passing, and reading definitions on my own was useful

Would improve8

Slightly more signposted exercises, but the open endedness was nice
The time management should have been better. I would have appreciated having more time for the exercises.
Some topics I've found unsatisfactory info on: theoretical interpretations of the emergence of the belief state geometry representation, its limits in the data generating process complexity given limited learning model dimensions (which was inquired about during the day), ablations (or some more sophisticated architectural variations) that are sufficient/necessary to break or weaken the belief state representation, whether belief states emerge only when trained on clearly HMMs or they are useful approximations of unknown or not clearly Markovian data generators, probably more. The topic seems to feed well into motivating RL optimality and universal prior related curriculum. To connect them, though, a more analytic or more quantitative understanding of how deep-learning in particular enables belief states to be represented should be instructive. It may actually have come up during the DL intro but slipped my attention. (I'm wondering if upon revisiting, or even first hearing about it later, today's material would be a useful anchor for capturing it? It seems likely.)
The schedule was a bit too ambitious: we ended up not reaching the last module. But as I said, that did keep us going so maybe that's not entirely a bad thing.
The other teachers gave us colab scaffolding, with access to solutions, when they wanted us to code something unfamiliar in 20 mins. Some of them also gave us clear Plan B’s, so if a few of us couldn’t keep pace with their vision for the day, we could fall back to doing/reading something easier/slower that was still relevant and useful.
I think some live coding of the theoretical concepts into visual applications like what we saw in the exercises would have been useful.
It would be great to hear a bit more at the start about the overlap between mechinterp and compmech so that it’s clear how these fields are connected but different or rather that compmech is a subfield of mechinterp
I think the timings were slightly squeezed in some places (the initial programming task, and the presentation), and the lecturer could be a bit quieter where the room is fairly small. I think highlighting the meaning of the non-invertible map A would be good, because it took me a bit of time to properly understand that the states were not redundant because the state would enable several token prediction, whereas A non-invertible means that for single token prediction the belief state doesn’t fully matter (in the suitable way). I think it took me a while to believe in the idea of studying HMMs as showing anything useful, so maybe highlighting that they’re fairly broad might be good (but maybe this was done). I never properly understood GHMMs, but maybe it’s not important for a 1 day exploration. I think the exercises maybe could cover the theory a bit more, and having a handout/notes in LaTeX with these would be really nice (if it’s worth the time to create).

Other thoughts1

Thanks! I found it very interesting

Data Attribution 5 runs 92 responses, 119 written

for pasting into Claude

October 2026 Thu 22 Oct Charles Renshaw-Whitman 24 responses energy 3.5/5

Rated against what each person expected. 0 is as expected.

Overall -0.50 below expected

Content -0.27 below expected

Clarity -0.32 below expected

Motivation -0.14 as expected

Exercises -0.40 below expected

Readings +0.15 as expected

Other activities 0.00 as expected

Pace +0.09 about right

Difficulty -0.05 about right

Novelty +0.59 mostly new

Read what they wrote21

Worked well4

improvement on last lectures he has given as he gave somewhat of an overview before reading, bu could have given a more structured lecture
Nothing, really. Maybe the videos Charles spent 10 million tokens on are insightful, but he did not really present them in a way that raised our expectations much.
I think Charles is really interesting and always appreciate his takes.
I liked having lots of breakout time to walk through the derivations myself

Would improve13

Not much, maybe some dedicated slides?
Slides
The notes are well organized, but the lecture was not. Charles believe is self-directed learning too much for it to have worked.
I liked the initial lectures on the topic, but near the end some more guidance wouldve nice.
The lectures are a bit unstructured and unorganized. Improving the presentation and organization would definitely helps a lot for understanding the materials better.
Today felt like a wasted day. The other day that felt like this was the first day Charles was "teaching". I was optimistic to hear that he was going to alternate between lectures and exercises today, but unfortunately my optimism faded away within one hour, after Charles refused to provide informative answers to questions (rather than quips), failed to reproduce derivations on the board, and more generally had clearly not prepared a coherent presentation of the material. Most importantly, he kept undermining the material he was teaching as he was teaching it. The Intensive is described on the website as "A four-week, full-time taught course". I don't see why you would entrust so much of the course to someone who seems completely unwilling to provide good pedagogy. The course page for today also has pretty bad notation (β used both as the datapoint weights and as the inverse temperature).
I would add a bit more of structure. I like a lot the idea of learning by reading relevant literature, this was very cool! However, I would have appreciate a clearer program to follow, for instance to coordinate 30 min of reading and 15 min of discussion, or something on these lines.
Prepare a lecture, not reading a doc
I think data attribution can be really cool, and Charles is a fun and intriguing guy who clearly knows his stuff cold, but I did not learn much because of two reasons. First, I felt like we did not get enough time yesterday to really work through our project, and I found myself wanting to go back to refining it further, and more generally I felt like _doing_ something instead of another day of _learning_. Second, Charles seems like he does not enjoy teaching very much, and does not appreciate how important context is before moving straight to the math in a way that suggests that it's mostly trivial stuff that we would learn better on our own. And I'm sure that's the case for him, but that's just not the way that most people learn or get passionate about a new subject. So what would I improve: allow for more time for practical work (which many people chose to take spontaneously now), and get teachers to be passionate about teaching.
I think it is interesting in and of itself, but it seemed like data attribution is not generally used by frontier labs to improve models, which I thought was unfortunate. It wasn't entirely clear what place it has in larger machine learning research.
Thank you for taking into account the previous suggestion to include lectures. They helped me a lot to get the overview of topic. The topic itself was bit far from my expertise. Despite that I got an impression of why this is done. I would include few coding exercise too.
I think Charles did a good job showing us where the topic fits into the bigger picture
I would have spent a little more time on notation in the beginning. The lecture was pretty rapid on math and it made it hard to follow without grasping the notation.

Changed my mind3

I did not really know about this topic beforehand.
I think model susceptibility under data perturbation is a fascinating idea. Wish I had more time to look into it.
This really got me to think more rigorously about how to do attribution at different levels/scales within models and systems. However, I think it was a bit general and would have appreciated seeing how it gets used in practice to make the connection to AI safety clearer.

Other thoughts1

Let me be the person who raises some uncomfortable criticism. I believe the lecturer is knowledgeable and competent, but I would like to share a few concerns for the sake of improving future programs, even at the risk of becoming a less favorable fellowship candidate. 1. In a professional academic setting, I found the frequent use of F-words and certain jokes inappropriate. They made me uncomfortable and affected my trust in the learning environment. I spoke with a few others who shared similar concerns to varying degrees. 2. While no doubt that the lecturer clearly understood the physics, I struggled to see how the theoretical material connected to AI applications. This may partly reflect my own limited background in physics, but more concrete examples, motivation, and explicit connections to AI research would help participants from different disciplines stay engaged. 3. Some responses to participants' questions felt dismissive. Rather than carefully understanding and addressing the questions, the lecturer sometimes responded with jokes or answers that did not sufficiently address the point. I believe respectful engagement with questions is especially important in an interdisciplinary program. To be fair, I attended only the morning session and spent the afternoon working on our own project. My observations are therefore limited, and later sessions may have been better. I take this program seriously and want to make the most of every minute here. I have no personal disrespect for the lecturer, and my criticism is not about his expertise but about the learning experience. I am sharing this because I genuinely value the program and hope the feedback can help improve it.

September 2026 Thu 1 Oct Louis Jaburi 15 responses energy 3.5/5

Rated against what each person expected. 0 is as expected.

Overall +0.14 as expected

Content +0.36 better than expected

Clarity +0.07 as expected

Motivation +0.43 better than expected

Exercises 0.00 as expected

Readings +0.38 better than expected

Other activities 0.00 as expected

Pace +0.14 about right

Difficulty -0.07 about right

Novelty +1.14 mostly new

Read what they wrote23

Worked well6

Speakers were clear.
The materials are very clean
Paper selection -- just incredible. As well as exercises and concepts. Very physics-coded
the exercises were instructive and let us go over the main concepts.
I think the lecture notes were written well and the material is compelling. I just would have wanted more time to think about these ideas!
Unifying influence functions with singular learning theory.

Would improve11

Share slides in during presentation or in advance so they can be annotated. Liked the chain diagram in presentation Feedback on Notes In section 1.4, is xi input and zi output, what is d_p, what is supposed to be p ? Missing y in “This question is typicall split into two parts:” in notes Some of the equations at the start of section 2 are not labelled.
It felt like there was actually no teaching and we had to read everything ourselves. Might be related to the fact that the speaker was online.
Interesting, but more lectures would have been better. Materials can be read at home.
Actually, a pretty nice day
I guess it was an exceptional situation today, so I don't think I could give really good feedback
In the notes, the total loss is sometimes averaged? Needs to be made more coherent maybe?
The notes were generally quite good and readable. I suppose I would've liked them to position Section 2.2 and the PBO more clearly as "here are some ad-hoc concessions practitioners make to reality" rather than "here is the correct and mathematically justified fix to a technical problem". Also, I thought unrolling was substantially more interesting than influence functions, so I might've liked to spend more time on it.
I think everything was pretty clear, and we got a good overview from the lectures but perhaps could have gone into more depth into a few things. This was understandably made more difficult by things being cut short.
Scaling limits of Hessian-inverse approximations.
make online lectures possible to listen on people's own device: the lecture area is too crowded and the screen is too small
I would have liked to learn more about the material and was sorry that this material was on a day that's been cut short

Changed my mind3

n/a
This might be the most important topic there is. From unlearning and ablation to narrow misalignment. Albeit it does run a risk of becoming needlessly mathematized, losing connection with reality
Attribution in deep models breaks down exactly at singularities.

Other thoughts3

n/a
Forgot to mention about the previous day with Ashe -- choosing our own agenda was awesome. In fact, it did make me think that Logical Induction might be a promising approach.
Clear, elegant math and great presentation.

August 2026 Wed 19 Aug Louis Jaburi 26 responses energy 3.4/5

Rated against what each person expected. 0 is as expected.

Overall +0.54 better than expected

Content +0.54 better than expected

Clarity +0.19 as expected

Motivation +0.69 better than expected

Exercises +0.28 better than expected

Readings +0.30 better than expected

Pace +0.12 about right

Difficulty +0.23 about right

Novelty +1.12 mostly new

Read what they wrote28

Worked well10

The lecture notes were amazing. Generally the way today was structured was really good, I liked being able to revisit previous chapter and the explanations in depth
The notes were _really_ great. Again, I was too low energy to keep up with them at the intended pace, but for me this sort of material is a really effective way to learn.
Great reading
Exercises and readings were appropriate
The exercises!
I think Lois is by far the best lecturer we have had. Better than the others at getting people to participate; good at motivating how this is related to safety. I liked that he used the marker and paper to lecture rather than a power point filled with bullet points.
I never heard of this before, great. Great how this, SLT and Training dynamics essentially use the same notation and build on a sort of "toy models of DL theory" kind of concepts (a little bit less similar notation on training dynamics) but in general seeing this "trend" of work is super great.
I really like today's format. I think the pacing is also good. The selected readings were also relevant and useful. Sorry should have submitted this earlier in the day.
The lecture notes were the best resource and I really liked it as a pure mathematician. It helped me gain a good formal intuition about what the data attribution was and what the three suggested approaches looked like in math. Furthermore, starting the day with the husband killer AI discussion and brainstorming how to fix the problem led naturally to the concept of data attribution broadly (hmm something bad must be in the data cause where else would it get such crazy takes?). This informal inkling was then very nicely complimented by the formal maths that formalised the concept and suggested ways to compute the $\pi$ function from data thru parameters all the way to $\mathbb{R}$. Definitely enjoyed!
- having written notes was very useful - content-wise, the discussion about unrolling

Would improve11

Whiteboard was a bit hard to read, the connection to the causality shapley values could be made more clear
I didn't follow the discussion at the very beginning of the day. It might've been better to postpone that until after some of the reading/slides. Or alternatively, maybe I was just too tired.
this was great! I would just make more examples of how it has actually been implemented
readings first then explanations, since we can spend more time to understand the notation
The morning discussion was too dragged out. I think the exercises should've had more connective tissue with the ML.
I feel like it was good to have a small group of various backgrounds to advance on the problem.
I would like to know what happens after you attribute the data. There's a lot of evil on the internet so removing all of it before retraining seems difficult, and anyways it seems counterproductive for a model to not understand evil. I get the point for something like removing backdoors but I don't get other use cases. Maybe it would be more appropriate to make this 1/2 a day, since the data attribution subfield seems a little premature, and teaching 3 premature ways to do data attribution didn't feel more helpful than teaching 1.
There is too little vegan food (yesterday as well) so later in the day I become extremely hungry and it is hard to concentrate, it doesn't help to just eat the snacks in the kitchen either. The air quality in Cantor room is bad and also makes it hard to concentrate. I spend a lot of time to try to get a good intuition for what is going on, I barely got to the questions before it was time to reconvene. I might just have a low learning rate though.
During the morning we spent more than an hour and a half in a discussion in which people were guessing intuitions. Maybe people like this kind of stuff but I'd just preffer to have content. This time I think could for sure be used for following the lecture notes more thoroughly - again, those are quite good. A general trend in the intensive is that lecture notes tend to be quite good, we did like 30% of them during the day but I wish we did more. On the other hand, we spend time with hand waving discussions and sometimes classes (today the class was reasonable apart from the discussion in the morning, but in some other days classes took like 3h when should take 30-40min, like SLT and CompMech day). So yeah, I'd double down on lecture notes and I feel this would improve the understanding of everyone. Instead of doing a handful of exercises, give us the day to study the whole thing under supervision from professors there to help. One failure mode for the lecture notes is that if someone study them with the goal of "solving the exercise fast" (which happens here due to lack of time because time was used to something else), it's possible to solve the exercises mechanically because they're not super hard. If we had the whole day and were supposed to study ALL the notes - in a "I'm going to grind all these materials cover to cover 100% them", I think there would be enough time to actually stop some minutes to reflect about the meaning of the exercise and this would also be improved. I hope this was not a super confusing stream of thoughts, but yeah, I feel like iliad have the hardest part of a very good program figured out - the actual lecture notes - but then these extra activities kind of makes things worse
This was a good video when I learned about Shapley Value: https://www.youtube.com/watch?v=NBg7YirBTN8
I think sometimes the handwritten on the paper board / spoken parts of the day were a bit hard to follow because the main resource was a set of lecture notes. Sometimes these would be on the projectors and that made them impossible to read.

Changed my mind7

Not so much today specifically, but over the course of the last several days I've gained a greater appreciation for empirical alignment work, even if it's not what I personally am instinctually excited to work on.
more ideas for attributing data to behavior
No
I never knew this whole field existed so that was important information to get
This seems to be a lot more useful than anything else we have gone through.
Panel in the evening was super useful.
The extent of worrying phenomena (e.g. subliminal learning) that are only partially explained by theoretical results

June 2026 Thu 2 Jul Louis Jaburi 13 responses energy 3.5/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.8 of 5

Teaching and format 4.5 of 5

Difficulty match 2.9 of 5

Energy 3.5 of 5

Read what they wrote11

In their own words10

The talks were well-done and the day had a nice narrative structure. The selected papers complemented the day nicely. I found Louis' insights about the field in general as well as his specific research areas particularly useful. I was a bit tired in the second half of the day, but even still, I feel like I didn't have quite enough time with the exercises.
Good use of blackboard, calm explanations, very knowledgeable. I especially liked the brainstorming at the beginning and the insistence there to wait until we come up with more interesting ideas. minor errata in the script: defintion → definition typicall → typically
Very good lectures, very good structure of the material, and very good actual material such as the curated list of papers, the exercises, and the lecture notes (these were specially very good). In all, one of the best days of the program.
I liked the talk and exercises, and the diverse set of papers illustrating different data and safety-related phenomena (deep ignorance, subliminal learning, poisoning, ...) gave a good overview of the problems the field is trying to address. I found the comparison between mech interp as "causal theory of a forward pass" and data attribution as "causal theory of a training run" a nice idea, and was surprised how far this analogy could be taken.
The content was really cool to learn about. I was a bit slow at the math so I didn't make much progress on the exercises, and I would have liked a bit more time, which is why i put 4 for the difficulty match. not a huge issue tho since there are solutions at the end which i can reference later if needed. overall seemed like so much to cover in one day, and I think it would be worthwhile to make more time for topics like this.
- I loved the morning discussion thinking about what techniques we might use for data attribution
I think it was a fairly interesting topic. At times quite technical without really having much space or possibility of making it more intuitive and having (toy) examples. If it can be made a bit more hands on with a practical exercise or demonstration, that would be great but I guess you can't do that with every topic. Louis was a good lecturer and did a good job of breaking down the concepts. I also liked the old-school-blackboard lecture.
I find the topic interesting
Good discussions at the start of the day which was useful for intuition, but that meant not as much time was spent on the meat of the theory and exercises.
I could only make half of the day. In the afternoon the lecture notes were good, the context switching between exercises, lecture and reading was well-paced and it was overall very informative, if a little rushed.

Other thoughts1

today's session was good.

April 2026 Sat 18 Apr Louis Jaburi 14 responses energy 3.5/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.1 of 5

Teaching and format 3.9 of 5

Difficulty match 3.1 of 5

Energy 3.5 of 5

Read what they wrote36

Worked well13

People made jokes in a collegial mood
I think I have a reasonable overview of the research directions in data attribution now, and the main methods for this. Seeing the results of papers in the class was very nice.
1. The QR code for questions were such a good idea. although it was underutilized I think we should keep this concept for other models moving forward. 2. The instructor showing relevant papers was very needed. 3. Good context behind what's the use of this research agenda and its relevance to both AI safety and other concepts we've built upon (SLT).
Lot of time for questions, lecture notes were extremely good
Good presentations
The lecture notes are great, and the number of topics/sections was appropriate for the amount of time we had.
The notes are informative and useful
The morning of lectures and afternoon for reading and exercise worked well. Also being told to use Claude to work through some of the "less interesting maths" was quite useful, having the validation that we can just move on if we get the idea is nice.
Lecture is very good and setting times ppl have to be back on discord made class flow better than normal
The notes were extremely detailed and it was useful to see everything spelled out so thoroughly
Good notes (great intro to important parts of the topic; technically a bit dense but good explanations throughout) and group discussions
The walk! I wasn't expecting it would benefit me, since i didn't consider energy level to be a problem in the afternoons, rather a diffusion of focus, and it ended up helping with that. Of course it's just 1 experiment, so no conclusions yet :) Also the group cohesion went OK, i felt we were more dispersed on other days, which i didn't enjoy -- but there's still room for improvement imho...
Breaks, mix of reading and exercises, regular and scheduled check ins

What they took away13

The connection to optimal transport was unexpected and cool.
The idea of doing unfolding with automatic differentiation is really nice, and the ideas for future research directions are nice (e.g. nothing known in RL)
Unrolling, influence function, bayesian influence function
Just generally getting an idea of what people are working on in data attribution! It was almost entirely new but super useful
The high-level considerations of data attribution
The general framework for data attribution: assigning weights to the datapoints, computing a relevant model configuration/checkpoint, evaluating for some property of interest, and then backpropagate to measure the influence of each datapoint. It was interesting to see different examples of this.
The unrolling seems interesting and useful, especially MAGIC in that it provides information about the causal impact of a datapoint throughout training
How we can skip lots of computation by looking directly at the weights and evaluating them.
Learning about meta gradient descent this was huge for me really interesting
The fact that classical IF are approximating proximal Bregman response function and not leave-one-out loss was very nice to see. Also the unrolling content was good + something I hadn't seen before.
Current frontier of the field of data attribution
The entire idea of influence functions was unfamiliar to me, although i have heard the term, this was an introduction to me. It was inspiring, albeit partly due to dissatisfaction with the underdeveloped theory (and that's probably the way it should be).
Data attribution is messy, there are some approaches in development of varying rigorous justification, there are some industry standard/practical methods

Would improve10

I fell behind and never caught up. I felt like I didn’t have the prereqs for this vision (pace, scope) of the day, and my time would have been better spent abandoning the curriculum and re-installing those dependencies than lumbering in the wake of the instructor’s train of thought.
I think I found the notes relatively hard to follow, where I think I'm not especially strong on multivariable differentiation intuition, and I think the notes were some intermediate of being very precise and being an overview, so probably more clearly defining everything said, or being clear where deliberately not, would help. It feels like the overall ideas are quite straightforward, then the implementation or approximation methods are much less so, and possibly the balance of time on the methods vs experiments + results from it was too much of the former.
1. Should highlight direct implication/most relevant results from research papers, and any remaining bottleneck questions left unfulfilled with current research to guide research directions. 2. What effect does this have outside of AI safety? Would digesting on this accelerate any capability potential work?
Im not sure that the matrix calculus details are useful.
We spent far too much time on influence functions and BIS which are in fact not very useful thw former because they don’t apply to NNs and the latter because SGLD doesn’t give reliable estimates of the quantity of interest.
I would add more regular small breaks in the afternoon.
It was alright for me but I think there are a few people today for which the mathematical density of the notes was probably prohibitive based on my experience with others in the past couple of weeks?
A coding exercise where we calculate influence functions in a trained model might be nice to include.
Q n A seemed divergent. It is a highly subjective opinion, that's true. I may think so because the preparations (planned timeslot and askqueue) made it seem like it would be somehow more structured, focused and/or better scoped; also i often try to listen in on other's questions and discussions with the TA outside the class setting and today this wasn't feasible in multiple cases. In any case, today i became mindful that synthesizing different lines of inquiry that different participants follow can be an aspect in which some days' teaching and format can work better or worse. (Now i'll see this as an even bigger challenge for tomorrow's "cross-pollination sessions" -- although it may turn out to be irrelevant.)
Document could use more refining/some more literature analysis/practical exercises

Debate and Provably Safe AI 5 runs 107 responses, 141 written

for pasting into Claude

October 2026 Mon 26 Oct Aden Power 33 responses energy 4.0/5

Rated against what each person expected. 0 is as expected.

Overall +0.42 better than expected

Content +0.42 better than expected

Clarity -0.10 as expected

Motivation +0.32 better than expected

Exercises -0.03 as expected

Readings +0.11 as expected

Other activities 0.00 as expected

Pace +0.10 about right

Difficulty +0.26 too hard

Novelty +1.00 mostly new

Read what they wrote41

Worked well13

Aden is very engaging and knowledgable.
I liked the discussion on complexity theory, but I would have liked to gain more intuition from it.
Small post-reading debate among fellow students
liked the presentation and the exercises.
The handout was nice to reinforce the more difficult proofs
exercises and discussion
The exercises and demos were very illustrative
Today felt very interactive!
I thought Aden's style of presentation was very good, the exercises felt relevant. The pacing was perfect imo.
The reading
The day was well structured and planned.
Aden's enthusiasm and willingness to teach! I really liked the exercise of reading the papers and recording at each step how our beliefs changed. It was a simple, almost gimmicky thing but it helped frame the reading!
The theory was very well explained

Would improve18

I thought the lectures were good and the topic was interesting--I think I had trouble following just because my energy was low. But I am super grateful for the material and will look back on it at a later date.
The mathematical formalism was interesting and I'm glad I saw it, but I'm not sure if it helped me to better understand how agent debate works. This picture gave me a different understanding of what debate is (in terms of complexity theory) but this didn't change my empirical understanding of it. I'm new to the topic though, so I might have missed something.
It was quite excellent! Perhaps speak a bit more slowly with less filler words like "uhm" and "like", but overall I like Aden a lot.
The lectures are overall quite nice. Probably explaining at a slower pace would help for those of us who are not familiar with the topics.
A little more details on how to actually implement debate in training
The lecture was good. The content gave a good overview of the topic.
I'd trim away the proofs, use more figures, and begin the lecture with the Alignment motivation.
The connection to how this would be implemented in practice or what are the practical implications could have been clearer. But still very interesting topics.
The lecturer made some mistakes that confused people
A bit less time introducing complexity theory, a bit more time on later topics, especially the prover-estimator debate towards the end.
Fix a couple mistakes in the slides that created confusion
Maybe a bit more emphasis on the connection between theory and empirics for debate would have been helpful today.
Only a small wobble on notation
Maybe some more preparation for the presentation
I wish the topics were better split between the theory and empirics of debate as a means for scalable oversight. The theory is just so far from how it's actually implemented that it is jarring to go back and forth.
Some of notation was confusing as presented, but it was eventually clarified in the QnA.
Despite his enthusiasm and passion, Aden seemed a bit out of his depth on this day. I would suggest improving the slides and the presentation in general by doing a more standard definition-example-theorem-proof kind of presentation (many slides didn't have definitions, theorems didn't list assumptions clearly etc) and then of course prepare more what is going to be said. I would suggest presenting and thinking of examples of the different classes and present them immediately. There _must_ be traditional and effective ways of presenting this material! Also I would totally introduce and rely on the Turing machine as the backbone of all this
The theory was nice, but the connections to how debate is implemented in practice were not fully clear to me, i.e. how this formalism translates to the way it is done in practice/how these are useful.

Changed my mind10

I am more amenable to thinking debate is a viable option.
I did not realise there were so many complexity classes
Didn't know about the debate approach to safety; seems immediately-applicable.
complexity theory has interesting points on AI safety, which is cool
I am really excited about ai safety via debate+scalable oversight and would love to pursue it. I have a strong background in computability theory, decision theory, and logic so this approach to ai safety seems very interesting to me
I am a bit more open to the idea that debate and estimation have a place in AI Safety rather than being pipe dreams.
I think debate is a more valuable research area than previous.
Debate is more important than I thought
I am a little less skeptical about debate working as an AI Safety technique, though not yet convinced.
i didn't know debate had such hardcore CS-backbone! Of the hard maths we saw so far, it seems the one with the clearest application to scaling and safety in practice (even though the connection between theory and practice is still kinda loose)

September 2026 Mon 28 Sep Stephan Wäldchen 26 responses energy 3.3/5

Called “Debate” that cohort.

Rated against what each person expected. 0 is as expected.

Overall -0.04 as expected

Content 0.00 as expected

Clarity -0.21 as expected

Motivation +0.08 as expected

Exercises -0.43 below expected

Readings 0.00 as expected

Other activities -0.11 as expected

Pace -0.04 about right

Difficulty +0.08 about right

Novelty +1.21 mostly new

Read what they wrote36

Worked well8

The game in the beginning -- incredible warm-up, where we were looking for mistakes / errors in simple proofs. This is how most lectures should be starting honestly. Nice balance between exercises, lectures, and papers. Going through exercises together! This was the first time it was done. Delegating it to going through the solutions perhaps also works, but definitely not as good. One of the best days I had indeed -- very applicable math, which finally goes into computational and memory complexity from applicable perspective -- our agents no longer possess infinite compute and memory. This is a great start. As if all my grudges from previous days (where fields had unrealistic assumptions) were finally addressed. And overall the day was very practical, grounded, varied, and at the same time fundamental. Loved it from beginning to end. Lecture on modeling beliefs via graphs -- good, very good, very empirical and theoretical at the same time, Econophysics vibes. Also, Stephan is great lecturer, excellent in fact. Or maybe I am biased to the topic, but really, I was engaged more than most of days.
Speaker was good.
The reading and discussing was separate but also intertwined well.
Reading by our own
Nice choice of reading!
I think this is a really interesting field and one I hope to revisit outside of the Intensive hours - I can definitely see myself working on this further, and I might try to write about it for my project proposal.
Complexity-theoretic framing of debate and interactive verification.
Reading papers and discussing results as a group was quite helpful.kept things very concrete in the second part of the day . Talking about what the curent state of research is.

Would improve18

I would perhaps explain first the idea of applying debate to AI oversight (namely the last presentation). I also found the latter parts of the lecture (related to statistical applications of debate) more interesting than the deterministic debates.
We really need a paper-reading workshop. We had like 30-40 mins per papers, and that time flies. Otherwise can't point out much, I liked the day.
Give more context for the problem at hand.
Some of the reading was not very high quality.
I think it would've been helpful to have more directed readings. Given the limited time available, I'd prefer you offer some hints as to what parts of each paper we should focus on, what information we should try to get out of the paper, or why the paper is important or surprising. The exercises were fine but somewhat redundant with what had been explained in lecture. I'm not sure if it's worth changing them though, interesting problems are fun but not essential for learning.
The narrative is a bit unstructured and hard to follow, and it seems the lecturer wasn't aware of the material that already was taught during the course. It's not a significant problem, but as the lecture is on the last week, contextualizing it better is definitely possible. Some results from AIXI, abstractions etc. are directly applicable to debate.
I prefer slides, but this is something personal
Good lecture day, thanks. I would suggest much clearer planning: for example, trying to adhere to the schedule, stating time allocated for reading. Also, more time for reading was needed, in my opinion. It felt a bit wasteful, after a break, to recall everyone in the main lecture area, waiting for them to be seated, only to announce we had to go back to our seats and read.
I think the content was generally good. The exercises at the top of the day didn't seem very related. I found the content a little demanding today, so in the afternoon, I preferred to read through the notes/papers myself rather than listening to the lectures, where I was starting to get a little lost.
I think doing exercises together on the screen is not super useful, since the amount of time people need to think through them can vary a lot, depending on background. Introducing the AI case a little bit earlier, even just a few sentences during the introduction could be nice as well.
My main criticism of today has to do with (what I felt to be) the lack of lecture time: the lecture parts could have been longer and more formal (but still comprehensive); the little formal content was in the reading/exercises, which for me is not preferable in comparison to the average modus operandi of the previous weeks. I'm not being exhaustive about the things I liked, but I definitely have to emphasise how the way papers were discussed (all together and with slides) is a huge improvement compared to previous days.
The presentation is a bit too math/cs, I think that currently applied debate is different and doesn't use all that machinery. So it may be restructured, e.g. going with practicalities first, and then give theoretical underpinnings. Also, using the latest papers might not be the best idea because a lot of them are slop and better look for some battle-tested blogposts even (so the reading is not well chosen).
I would improve the choice of papers for the first session. They were rather bad papers and were not worth my time.
More motivation at the start, some time to do the exercises.
More time on prover-estimator debate variants.
I found the notes on the complexity classes a bit strange. They try to be introductory but at the same time have to make compromises and present more advanced topics, making it overall a bit confusing. I think having slightly longer notes for the sake of complteness would be very nice.
I don't see how the experimental results connect to the theory except that there's some justification provided by the theory to do debate.
The first part of the day could benefit from having slides. I found that the discussion about Turing machines was a bit confusing. The exercises and notes would benefit from clearer language and a few reminders about complexity theory .

Changed my mind6

That Debate might be extremely effective. I recalled that it incentivizes each side to pull the most extreme arguments, which definitely helps. Also, modeling beliefs via graphs was mind-blowing -- simple yet realistic and effective. We need more theories like this.
n/a
Slightly more optimistic about usefulness of debate
learned about AISI
Debate offers a much more credible path to scalable oversight than pure distillation.
I found the contents very appealing for research.

Other thoughts4

I will have a few interviews tomorrow and will thus likely go to Honeywell to take them, but it will be 1h between 3pm and 4pm, thus shouldn't be too distracting.
n/a
I had a terrible day for personal reasons and it was very hard for me to pay attention today. The feedback reflects on my experience but it is by no means objective.
Super clear exposition of AISI safety cases.

August 2026 Wed 26 Aug Stephan Wäldchen 25 responses energy 3.4/5

Called “Debate” that cohort.

Rated against what each person expected. 0 is as expected.

Overall +0.80 better than expected

Content +0.80 better than expected

Clarity +0.48 better than expected

Motivation +0.80 better than expected

Exercises +0.15 as expected

Readings +0.44 better than expected

Other activities +0.29 better than expected

Pace +0.16 about right

Difficulty +0.36 too hard

Novelty +1.44 mostly new

Read what they wrote27

Worked well8

Gong usage was pretty good although I was late to one because I don't think it was hit (maybe I didn't hear it?).
group readings and exercises
Rigorous schedule
The exercises were great and not hard to follow and the structure of today was quite amazing. Stephan also communicate well and bery structure. Honestly one of the best teaching day so far!
Great instruction, seemed very knowledgeable and enthusiastic
I really liked that Stephan actually made good use of the day (few breaks, lots of material)
I think the method of having people go up and present the paper is great way to communicate the key ideas from multiple papers without having everyone read all of them.
The exercises were helpful and the content was great :D (and it was nice to have a whiteboard lecture)

Would improve11

I think Stephan should've went slower on some things (some of my friends were pretty lost). Also, maybe making the big picture of debate clearer would help. Some of the exercises were kind of trivial. I didn't learn much from the readings.
I found it quite hard to understand how the toy models of debate and protocols are scaled up to frontier model levels, so a bit more words on how this approach is expected to play out for alignment would've been good.
This was one of the days I was most excited about. In the end we had like 2 sessions of 15min to read the lecture notes (which are the things that are actually detailed with theory of debate), and I guess the teacher expected us to get all the details in these 15min? I just couldn't. The other papers were understandable and the classes were fine, but I just don't think they would do a better job than good lecture notes. I guess the main suggestion is improve these lecture notes so they are a little bit more self contained (for ex., I didn't know all the definitions of complexity spaces used in the notes, and the notes just gave some handwaviness definition of them, but if you're actually proving stuff about something the object should be clear).
spend more time on group exercises
Today was amazing! I'd definitely change the morning readings (AI researcher). I felt the 3 papers are in the lower quality end and they didn't really fit into the "trifecta" framework
This is hard to say because Stephan is a wonderful instructor and the course structure is laid out really well. I don't know if I'm biased since I just realized that Debate is using a lot of methods from game theory and computational theory of mind which I'm quite familiar with. The exercises were great and not hard to follow and the structure of today was quite amazing. Honestly one of the best teaching day so far!
there were some things squeezed in such as the final aisi slides, didn't follow them entirely. also 10 mins for the exercises isn't enough
I think I'd have preferred getting a better idea of what *work* in the field actually looks like, and see a bit more the connection between theory & practice. I think the discussions after the readings could have been improved - hearing others discuss papers that I haven't read isn't super interesting, especially because no one really wanted to. I think from a pedagogical POV, Stephan should just summarize the papers.
I think today was a great topic and I really enjoyed it, but I think many people were (understandably) distracted by trying to fill out the interview for Garrett. Maybe I would try to release it as a timed test or something in the future...? Maybe with people having the questions in advance a little to reduce stress (sorry I am worried my peers will hate me for suggesting this haha so I am trying to balance my suggestion in favour of what they might want me to say for their benefit)
I would have skipped the first reading session: At least the math Lean paper seemed to be summarizable in five minutes, so imho the time could have been used more efficiently for other things, but I understand that this might just be my preference
Very nit-picky, but I wasn't a big fan of the VeriPlan paper (but it was still useful to set the ground for debate)

Changed my mind6

Scalable oversight seems very promising
Debate has a super clear and strong theory of change
I now understand why people in Game Theory and MARL decided to work on this topic.
improved liking of debate
Today's safety agenda seemed the most promising out of all the things I have seen so far
Without having any knowledge at all debate just sounded weird to me. Seeing exponential complexity to polynomial was a great motivation. Its actually really interesting ! One of my favourite topics

Other thoughts2

A lot of self-play method has been used quite significantly in debate and I'd like to investigating debate a little more!
Interesting mix of activities today. Generally, I think it would be helpful if the days carried more descriptions in the calendar, eg what the prerequisites are expected, what topics will the day build on. That way we could do some preliminary reading or something. This topic was definitely super difficult to anyone who hasn't done any computation/complexity theory before.

June 2026 Tue 30 Jun Stephan Wäldchen 11 responses energy 3.3/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.5 of 5

Teaching and format 4.5 of 5

Difficulty match 2.7 of 5

Energy 3.3 of 5

Read what they wrote11

In their own words10

I really enjoyed the part about AIS via Debate, and I found the reading (Debating with More Persuasive...) very interesting. In general, Stephan gave a nice overview of the contents, withtout leaving out critiques and opinions–very honest. I also appreciated the reading list. I think the selected papers were pedagogical and helped me understand the concepts without overcomplicating them too much.
I appreciated how a well-posed safety case and empirics clearly accompanied the theoretical work we did. The content was well taught. I particularly enjoyed the exercise and obfuscated arguments sections. I found everything helpful except for the first paper reading + discussion. I don't think the setting needed more clarification. I would have appreciated a more technical reading here.
Provably safe AI is an interesting research direction, answering the practical question how we could trust outcomes of non-trustworthy AIs. While the provable framework with a given world model and verifier appears a bit limited in practice, the debate framework appears to me as a good practical approach to work towards AI alignment. The presentations, reading material and exercises worked well for me to get the subject across.
Good introduction and motivation
- I really loved the safety case. It is the level of detail that I want from a safety argument - I liked the breadth of topics covered - We didn't have any breaks. I would have really liked to have a 10 min break in the morning and another 10 min break in the afternoon - It would have been useful to receive some reminder of the pre-requisites so we could review a bit before the lesson. This was not an issue for me personally, I think I could follow from my previous knowledge of Turing machines, but it is probably useful for others.
Today's discussion format was the best in my opinion. In previous days in each group everyone read different papers and we tried to discuss all but today each group members read same paper so we are actually able to discuss and then all groups came together and we learned about each paper also the instructor commented on them which was also quite nice.
Very interactive, using the blackboard effectively. A bit more time to prove things on your own could have been useful. the quiz was quite good, the starting with the subtly wrong proofs set the scene very well.
I think it's an interesting, realistic research direction. I am not sure I understood the final slide presentation about the safety case. But the formal and semi-formal debate structures are interesting and the empirical results in the paper we read seem at least promising.
I found today not as interesting as my favorite topics so far, but the most convincing as an actual step for making AI safe. I thought the format was ok, but I would cut out the readings we did in the morning, since they did not add much to the idea introduced beforehand and they were boring.
One of the best days in terms of the format of the day. I think Stephan had scoped the sequence of tasks very well. In the morning, we started with a small lecture followed by paper reading in discussion groups. A discussion group is standard for most days but because we started with it and the time management was good, we got ample time to have all groups present their papers and have a Q&A - something we rarely got time for on other days. Throughout the day he adapted very well to how the energy in the room was, how much time there actually was for each task, and in general he had created a good mix of discussions, lectures and exercises - also keeping in mind the balance between covering the theoretical ideas behind Debate and the experimental studies. His lectures with more blackboard writing than Claude-generated slides were also quite engaging! In terms of improvements, maybe the experimental papers on debate could have had better prompts for discussion but that's a minor thing.

Other thoughts1

dug thru the ai debate repo, no stats code or judge verdicts, so numbers are kinda just trust me.

April 2026 Tue 28 Apr Stephan Wäldchen 12 responses energy 2.8/5

Called “Debate Protocols” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.5 of 5

Teaching and format 3.3 of 5

Difficulty match 3.1 of 5

Energy 2.8 of 5

Read what they wrote26

Worked well9

The content was intuitive. The mixture of presentation, reading, talk in the afternoon and finishing with a debate from two AIs was good to keep the energy up and the day engaging.
Initial presentation was good (but a little long) and discussion/criticism of literature across different groups
The formalisms of the debate protocols
Good diversity in teaching formats, and all parts were good !
The activities were varied and balanced, the scope seemed narrower than on other days and it likely worked in favor of understanding the material.
I liked the debate between Claude and ChatGPT, this was very interesting even if they both failed ultimately
The schedule was varied: identifying proof flaws, presentations, collective exercises, reading, and a debate demo. This diversity makes it easier to stay focussed.
The afternoon readings about debate in practice for AI safety, AISI talk, and live debate with LLM’s about AI safety (illustrating the protocol) were great.
The teaching was good + engaging, and I very much appreciated that in the afternoon we stepped away from toy/exactly solvable examples to proposals to how this will be implemented in practice.

What they took away9

Learning about the impartiality and the general role of humans as judges in AI interactions was very interesting.
Debate is 🤷‍♀️ interesting framework, but even the researchers can't comment on what works and doesn't so it seems like a hypothetical strategy
Cross examination
I didn't know about the Debate protocol and I found it super useful to know about this technique and the related criticism.
That debate actually has a well-formalized, mathematically rigorous definition, but it's highly non-trivial to map that onto how LLM safety can be improved by debate.
In principle I like this idea and find it useful but it seems to rest on many unrealistic assumptions so the most useful thing was to focus on the most specific instances. The most useful thing was this connection to why drilling down to semantic concepts is so hard and how debate in some circumscribed domains could massively benefit this. So in a way Debate seems useful to help us with progress on some fundamental issues that might then help us with safety, as opposed to being directly relevant to safety. In some ways the talk by Marie was more in this direction and that was useful.
Different formalization frameworks for debate (cross-examination, prover-estimator), and the idea of obfuscation as a debate strategy.
The idea of debate as an effective mechanism for probing safety of models — as well as a useful mechanism for me to get LLM’s to produce good arguments.
The afternoon talk with the sketched version of how advocates imagine this being used in practice was good to go through.

Would improve7

I would give more papers/article options as I thought hearing people's summarisations of what they read and their opinions (on the ones I had not read) was very insightful.
It seems as if this and a few other days could be squished into a single day of various strategies for alignment
Maybe describe the goal/motivation/setup more before diving in theoretical stuff.
The morning material and method started off seeming thought through and structured. There seemed to be some skepticism in some participants that the idea of debate is either useful in theoretical CS or in AIS in general. A germ of such a discussion started growing before lunch. Afterwards we jumped to the opinions on AIS via debate and I feel like we were unprepared to engage in reading or perceive that content in an informed, objective way due to little time commitment to the concept (and is connection to the morning's material). This made the early afternoon seem a waste of time, discussing things about which we couldn't be well educated. In my opinion laying the ground would also have helped in understanding the afternoon talk. I view the chatbot debate in the end superfluous and uninformative retrospectively.
I didn’t find it useful to work through the exercises together so fast as this, I feel like I didn’t learn much from them, would be better to have some time to try it for ourselves. I also found the material from the opening presentation wasn’t so easy to absorb today and the complexity theory details in the end didn’t seem so crucial for understanding debate yet we spent a long time on them, I think the emphasis could be shifted much more towards the question of what debate is useful or important without going so deep into the complexity theory.
A sketch of how debate might be used in practice for AI safety could have been useful before introducing the formal complexity-theory-based framework.
The more session was a bit confusing. The quantity of formal logic and turning computation in the session was intense. And I’m not sure altogether necessary? Do I need to know about polynomial length in order to implement a debate in practice?

Other thoughts1

Note on the low energy score, in case that went into assessing teaching quality, it was on me for not sleeping much the previous night. Also for this reason my focus was low for the late afternoon but I was a bit more energized towards the end.

Decision Theory 1 run 10 responses, 8 written

for pasting into Claude

June 2026 Fri 26 Jun Sung En (Daniel) Chiang 10 responses energy 3.6/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.0 of 5

Teaching and format 3.1 of 5

Difficulty match 2.8 of 5

Energy 3.6 of 5

Read what they wrote8

In their own words7

slides again too small, presentation is ok. the website is very cool.
Today was better than yesterday. Felt like some of the feedback had been taken into account. The structure of the second half of the day was also much better than yesterday with the discussion -> lecture -> tournament. The tournament was a fun exercise. One thing to still improve I suppose would be to make the lectures a little bit clearer as in the morning one it felt like there many unresolved doubts and a lot of back and forth without any agreement.
Understanding the differences between EDT, CDT and FDT was interesting, but - like the previous days - the topic feels a bit more GOFAI than applicable to currently more plausible paths to superintelligence. UDT and other some concepts appear a bit more esoteric. The game-theoretic part (commitment races etc) appears to be relevant to AI safety only in very specific and constructed scenarios (multiple AGI agents instead of a singularity) which I find interesting thought experiments, but of questionable usefulness.
The last exercise was quite nice and engaging The slides were just text and fontsizes were not readable at all I would be happy to learn more about game theory and then continue this would really be nice and interesting also more connected both for the context of today and in general with the other days
The lectures in the latter half of the day were better. I think the morning lectures should start with the examples and then develop the decision theories carefully. Clear definitions are important.
- There was too much text on the slides, and the text was too small - It would have been better to accompany definitions with examples more directly (e.g. EDT in the morning) - We very easily forgot about taking breaks
overall same feedback as other agent foundations today regarding teaching style, but the content was a bit more fun and I like that we get to do this tournament with the bots at the end. In general though, I think it might be nice if we spent fewer days on agent foundations, and more on SLT, mech interp.

Other thoughts1

fdt's detect the same function, is exactly my natural latents question, would love that thread in open source games.

Decision Theory → Reinforcement Learning 4 runs 104 responses, 142 written

for pasting into Claude

October 2026 Fri 9 Oct David Quarel 42 responses energy 4.0/5

Called “Decision Theory and RL” that cohort.

Rated against what each person expected. 0 is as expected.

Overall +0.46 better than expected

Content +0.51 better than expected

Clarity +0.32 better than expected

Motivation +0.12 as expected

Exercises +0.37 better than expected

Readings +0.29 better than expected

Other activities +0.06 as expected

Pace +0.54 too fast

Difficulty +0.22 about right

Novelty +0.66 mostly new

Read what they wrote55

Worked well17

The answers ti questions by the lecturer.
The notes introducing Solomonoff induction and AIXI were very well-written and good for internalizing the more difficult conceptual material
The Notebook in Colab. And the sarcastic approach of the teacher also worked well, although it sometimes made him slightly difficult to approach (somewhat afraid to ask stupid questions).
Got my first exposure to RL through the well designed Notebook.
I think the material and discussions with the lecturer were very useful.
The exercises were so well curated and detailed.
Loud voice, good preparation
The material touched on the core concepts of RL that are relevant to understand what is still being used/practiced today. So, after getting the basics I hope to read more and familiarize with the concepts more. Good foundation.
Building around the central objects of expected value.
The mathematically rigorous presentation helped me, and several of my colleagues, latch onto the ideas and formatively develop intuition. Most of the exercises were well-considered and helpful for advancing intuition.
interesting material, amazing energy and knowledge from the speaker
The topic and formalisms were new and different from those covered on previous days. I really enjoyed it, and even when the material was difficult, David was very good at explaining it again and answering questions.
I liked the rigour and the teaching style. Often was too fast, but having time to go through the material helped, and I believe with this much content you cannot go any slower. So no complaints.
The lectures were the best part, but I wish they'd been slower.
The materials is well organized and great for me.
mix of examples and theory and coding
The derivation exercises were helpful.

Would improve27

I think the most helpful improvement for both learning about AIXI and Reinforcement learning is: immediately after giving the definition of either of these, give a specific example. For example, it wasn't clear that, when talking about reinforcement learning, that the action one can take only has probabilistic distribution on what actually happens after, and I think this would have been cleared up if the two examples were moved first. Also, it seemed like these slides were originally made when reinforcement learning and AIXI were covered for a longer time; it might be good for the slides to be adjusted rather than to try to cover as much as possible.
Great
I liked it
Reduce the amount of material
Slower pace
More time explaining relevance of Solomonoff and AIXI results, especially in practice. Less time on mechanical exercises
Begin the day with the notebook exercises to motivate the mathematical content. Next, I would reduce the volume of the material covered, and focus on getting the main point across, rather than rushing through multiple large topics in a single day. Of course, this is all specific to my background, and probably doesn't make sense for everyone.
I think a lot of stuff was poorly motivated. I wish we would have talked about what was mentioned in the calendar.
I think today was great.
It was a bit much different stuff crammed into one day, maybe remove the last part, and focus on AIXI so one has more time to read through the later parts
Went a bit fast but I honestly given how short the program is this is better than prioritizing real-time understanding. I think I like the speed-run style much better as long as we're given the foundational exercises to work on in our own time to cover our bases.
It was great! Although I don't know how much of the first part of section D we really studied (which was interesting to me i.e. the more economics of rationality section)
Encourage the audience to filter the questions.
Too many materials covered in a short time. The math materials could be covered at a high level (focusing on intuition and proof sketches) instead of going too deep. The colab tutorial at the end is nice and deserves more time.
Make the instruction a bit slower.
I would welcome more time on this material could -- it seemed interesting, but rushed and cool stuff skipped. Introducing the example from the very end at the beginning could help ground the abstraction through the day.
maybe a bit fast at points
It felt a bit rushed, but it was okay as I knew most of it beforehand.
today was way too packed, with the subjects seemingly weakly related to each other.
It was really good and interesting content, and great teacher with great understanding of the content / field. Only issue on my side maybe was that the content links are sometimes a bit messy, as in some links in intensive-october.pages.dev, some in iliad-intensive.org, some in discord. It gets hard to follow. Would be nice to have a simple and compact way to get to all lecture resources. Applies to previous days as well.
I could not download one of the packages for the Google Colab.
This was probably too much material to fit in one day. I would have preferred to go through some of the theorems slowly in class, rather than going through them on my own afterwards. The notes had a lot of notation, which made it difficult to keep track. Lecturer was clearly very knowledgeable and good at presenting, but it was impossible to get through all the material comfortably within the time frame. The parts that I could follow were great.
I feel like, at times, I was struggling to to see the connection bewteen the specifics of decision theory, AIXI, and how it relates to AI safety and existential risk. One question I had going into this particular section was learning what the load bearing assumptions (inside of decision theory) are for existential risk due to AI. I do think I was told these assumptions... I just am not sure what they were. I mean, I feel like I got a sense that the needle is in the haystack of info today but I'm not sure where though. I am excited to look back at the info and learn from it.
nothing
This is best lecture day this week
Very good - personally, I would remove some of the quantity / amount of detail to be tracked. Not in a manner to dilute any one individual topic, but in a manner that - in principle - the material could have been presented at a black board - stripped down to its essentials. I emphasize that this is a personal take, constituting an improvement on an already good program.
The pacing was a bit fast, but perhpas the only way to fix that would be to assign pre readings or split this across 2 days.

Changed my mind9

I'm better able to understand how to study and bound AI behavior in general rather than just specific models
I wasn't aware that there is such deep theory behind RL
N/A
The importance of RL in the AI world.
AIXI seems very interesting!
Kolmogorov complexity is a very cool concept and seeing its use to formalize the often-cited 'occam's razor' concept was very interesting.
I am very interested in the AIXI/Solomonoff Induction research agenda.
I didn't think this topic was important/interesting before, and my mind has been changed.
I do think that I underestimated how much the AI community understood mis-generalization and I am curious to learn more about how they think about it.

Other thoughts2

N/A
The material was bit too dense for me. I knew about RL before in different context, not the theoretical basis of it. So, having everything on the same day felt like a lot. The theory session itself could have been a full day. The coding session, where we were given a code to play with was too short to really understand what is happening. Spending more time on that would have been more useful for someone like me who has more coding background than the math.

September 2026 Mon 14 Sep Fernando Rosas 38 responses energy 3.6/5

Called “Decision Theory → RL” that cohort.

Rated against what each person expected. 0 is as expected.

Overall +0.14 as expected

Content +0.28 better than expected

Clarity +0.08 as expected

Motivation +0.14 as expected

Exercises -0.25 below expected

Readings 0.00 as expected

Other activities -0.11 as expected

Pace -0.17 about right

Difficulty -0.22 about right

Novelty +0.67 mostly new

Read what they wrote50

Worked well17

- Grounding RL in preferences instead of just starting from rewards
The exercises with solutions. The formal introduction of the axioms with examples.
Less coding exercises: I think that this was a good choice given the material.
I liked the structure of today's lectures. Starting from the fewest assumptions possible and building up to something that has enough structure to prove results is very intuitive.
Exercises like Problem 2 where we actually did compute some stuff, plus a few other like Problem 1 which taught us how to derive the Bellmann equation -- helps invaluably.
I really appreciated the first lecture. I have read intro books on RL before but never delved into how the structure of preferences determines the reward function. Very interesting!
exercises! very fun to do some math
I really enjoyed the exercises.
We had all the materials sent beforehand
Lots of exercise time, Leo was very helpful in particular.
vibe-reading by my own the materials
I think the slides were structured well and there was a natural logical progression in the results being taught.
i liked the agency part surprisingly much
The logical structure of how stuff was presented made sense, we built well from one idea to another.
The initial discussion about what agency means, and how we should model it, and the introduction of the von Neumann framework.
not sure, spent the whole day being pretty confused about preference and utility
I thought the the connection between the fundamentals to the broad strokes of RL was illustrated with a lot of clarity.

Would improve22

I felt like introducing such practically relevant content in terms of axioms was a bit terse. I would've enjoyed it more if a more practical approach was taken to the content
More concrete examples
Good class today, but it felt too self driven. The lectures were relatively short (2.5 hrs) in total maybe, and the rest of the time was dedicated to independent exercises. I think it would have been more useful to maybe cover some relevant problems or applications as part of the lecture.
- Would have liked a paper or textbook chapter to go with the lecture (e.g. Sutton & Barto ch. 3 for the MDP part) to have something more formal to work from - More worked examples connecting the VNM axioms to reward / return / policy
The day was quite abstract and very proof-based rather than compute-based. I personally was quickly losing interest in the exercises as I did not see value in doing those proofs. Perhaps would be worth considering to have problem sets for differently inclined people: smth like Problem 2 but longer and much harder for applied mathematicians / theoretical physicists, some Colab notebooks with simulations / visualizations for more computationally-inclined students, and indeed proofs for pure mathematicians. Finally, just a selection of conceptual readings like LW posts or arxiv preprints to see what are those non-Euclidean preference axioms in practice for more philosophically inclined ones. Also, what I lacked was a more empowering example. Maybe it is hard or even intractable to provide one, but say showing how it works in say AlphaGo or another familiar system would be helpful to see the value in this. Also, the notation in problem sets: what is that infinity under the || metric?
The day was too sparse, too much of unstructured activities
This lecture was much better than last Friday's. It covered concrete set ups and I felt like I had some sort of closure after each of them. However, this feels quite disconnected from how people do RL, ie. it would've been useful to do some coding (we have covered the Arena material essentially so maybe presenting the corresponding coding exercises as alternatives to the theoretical exercises would be useful). Also, I am not too sure how RL is done in practice, ie. how does deep-RL work? How do we optimise policies in these cases, etc.
Add a third lecture - more material!
Liked the introduction to rewards in the first part, I was wondering if it can be compared to other approaches and discuss a bit more where it fails/what it can't capture.
maybe some discussion about the exercises as a group so we keep the energy more focused, but the exercises + lectures mix is personally so much more effective than lectures alone
I particularly enjoyed the second part. I kind of see the motivation behind the first part, but felt the assumptions where quite unrealistic, the notes were great, though.
I would have liked a few examples of contemporary research questions around the topics we covered today
I especially liked the first part related to introduction to utility functions and derivation of the common reward optimization framework from the common assumptions. This part was definitely valuable. Overall, I tend to believe that the core part of learning and teaching given that we all have LLMs right now is an ability to identify corner cases, limitations and point people to them. On the other hand, it might be useful to dedicate at least some time to Deep RL and some ideas and limitations that break in when we start dealing with finite datasets, off-policy learning and functional approximation instead of purely tabular settings.
Too much time for the exercises. I would have liked more explanation and discussion time.
I think maybe a few more slides on Q-learning, as it was a little more difficult but seems like quite an important aspect of this theory. I thought the exercises where quite easy apart from Ex. 6 which was quite tough.
i would have liked solutions for the exercises or discussion for what we should take away or should have taken away (afterwards)
I liked having the schedule for the day on previous days. One could also consider splitting up the lecture time more, since the exercises ran very closely along the lecture, so alternating more frequently good be helpful, as it would ensure that previously established concepts have been internalised before moving on. As it was, the wait between "you will do this in detail in the exercises" and the actual exercise was a bit long for later topics.
I found today's material pretty interesting; I've seen a lot of it before in some form, but this was a clearer and more complete presentation. However, there might be some room for improvement in pedagogicalness. To some extent it suffered from the classic "math's pedagogical curse": Equations and proofs were clear, but it took some work to read the underlying concepts/motivations out of the equations. For example, maybe some schematic discussion of MCTS would've helped give a clearer intuition for the Bellman equation? And certainly vNM/Dutch Books lend themselves to cute examples about drunkards and gamblers with irrational preferences. Generally any excuse to put a diagram on a slide full of equations sounds good to me.
- Maybe swap one of the analytical exercise sessions for a hands-on implementation of RL in a simple setting (e.g. where the set of states is small) - I would have liked to see more in detail when RL in the von Neumann framework fails, and what are the open problems
The language and notation in the exercises are difficult to understant. It's a pretty mathematical day as well, I'm not sure I saw the direct link to AI safety.
I would try to have more practical examples for the material. Plots trial systems, anything more practical to illustrate the math.
I was already familiar with most of the material and so I decided to read some papers from Simplex during the lectures/exercises.

Changed my mind9

I appreciated in particular the pre-lecture reading on decoupling beliefs/desires. That persuaded me that our theory of agents loses little generality for the common-sense meaning of agency.
That the entire field might be as misguided as Economics due to expected utility theory. The ASI most likely won't have consistent preferences because you don't need them, just like humans survive easily without them. No one Dutch books anyone or money pumps anyone because the effort to discover the inconsistent loop requires NP-hard computations, which no one ends up doing. Thus undermining the motivation behind the field and most justifications.
n/a
really curious now to learn how Q-learning was applied for the bot that plays starcraft
I discovered that even though RL is not the main technique frontier models rely on, it might be helpful to understand it because it is used in post training.
agency is much weirder than I realized, i find the vNM axioms much less obvious than before
RL really doesn't tell us much about alignment per se!
Did not get time to reflect on anything
RL can be worse than supervised learning in a lot of cases.

Other thoughts2

June 2026 Fri 12 Jun Fernando Rosas 14 responses energy 3.6/5

Called “Decision Theory and Reinforcement Learning” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.4 of 5

Teaching and format 4.4 of 5

Difficulty match 2.7 of 5

Energy 3.6 of 5

Read what they wrote13

In their own words12

I particularly enjoyed today's problems. I'd seen quite a few of the concepts before but Fernando provided great intuition to solidify them for me.
Two lectures worked well for me
More time was spent on lecture with slides than was necessary. I think the slides used today could be read independently without making the content less clear. I think doing something like slides and exercises at the same time--similar to what we did the second half of the SLT day, where we'd discuss for like 1-2 min what a section of the notes was about and then spend some time doing problems, and then repeat that cycle at regular intervals would be effective for todays content. I also personally had less interest in utility and RL to begin with, although given that it's included as a topic, I think it would be more fun to address this one from an empirical/engineering standpoint.
The more interactive format of the talks worked very well. good reading materials and exercises with good level of difficulty
Best part was deriving reward from preferences in one line: preference -> vNM utility -> reward -> MDP -> Bellman/policy iteration. The gamma=1/2 and policy-improvement exercises each carried a real idea. Would tie the end-of-deck agent taxonomy back to alignment more explicitly.
The session today was peak. Very good motivation, very good content, felt like I got to learn a lot. As a math and cs major with no physics background though, I found that stuff a little hard and not something I understood (the decomposition theorem and the energy relation with the utility stuff, formally). I will work on that, and if there are any suggestions, please do give them to me.
I was not very familiar with reinforcement learning and today's module really helped me grasp the basic concepts underlying this approach. The lecturer/speaker did a great job in conveying the main points of RL and I think I'll follow up to learn more. I also appreciated how the speaker organised the two blocks: the first more first-principles based, setting the context; the second had more practical implications, building upon the foundations of the first block. I wouldn't change much in my opinion.
Professor Rosas is an excellent lecturer and I greatly enjoyed today's sessions.
having 2 lectures kept me focused and also categorized the topics we covered and we also had a chance of solving exercises related with both sessions and exercises were quite beneficial which enhanced my understanding
- I liked the lectures - In particular, the content on the assumptions behind the reward hypothesis was very interesting - Personally, I feel like I was missing some reading to go more in depth - The exercises were useful to consolidate concepts, but the ideas were already quite clear from the lectures, so I would have preferred something to expand conceptual knowledge (e.g. think about RLHF)
I liked the afternoon session a little better. I appreciate the idea of trying to motivate (and critique) the standard assumptions of decision theory, but I think the result is a little too meta and confusing. It's hard to explain and critique something simultaneously. I would have appreciated either a little more precision in that first talk, or homogeneous simplifying assumptions. Instead, the notation there and in the notes/problems is inconsistent and confusing.
great lecture, especially the first part motivating reward, the second part was more standard RL but still good to have a refresher on that. The exercises are good, but it's hard to do all of them in the time, and I'm wondering if there's a better format (more structured) to do this more didactically, if explicitly splitting in groups or allocating certain time to exercises and certain time to discussions would be helpful. By default, people just do their own thing and that's good, but may be easy to get distracted, especially if there's no (peer) pressure to actually go through the exercises.

Other thoughts1

The in-session worked exercises were the highest-value part. More of that.

April 2026 Wed 15 Apr Fernando Rosas 10 responses energy 3.8/5

Called “Idealized Agency” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.5 of 5

Teaching and format 3.9 of 5

Difficulty match 3.1 of 5

Energy 3.8 of 5

Read what they wrote24

Worked well9

The first part of the day was my favourite part of the whole programme! The lecture was clear and engaging, and then I got to reread all the definitions and do the exercises at my own pace, understanding everything I was doing, occasionally (I think twice?) asking questions, without being pressured to speed up and skip things in order to be ready to rejoin the collective train of thought within 20 mins or to pair with someone who didn't want to go as slowly. This particular content, about AIXI, was/is not a priority for me in the context of my life outside of Iliad, but I expect that a month from now, or three months now, I will remember that content better than anything else I've learned in this program so far, because I was allowed to take the time to understand it.
Both lecturers were very clear, and I enjoyed the morning worksheet/exercises.
Quick lecture and recap of material
The first part on AIXI was taught efficiently.
Diverse combo of lectures, problems, discussion
I liked the balance between lectures and working on the exercises
David's lecture was good, exercises are nice for solidifying well, and Fernando's lecture conveyed the ideas of the area of studying the topic nicely
Good presentations !
Lecture from Fernando today was really great

What they took away9

in the past talking about Bayesian paradigms has felt a bit like switching briefly into a foreign language I haven't formally studied, knowing I won't sound fluent, and just hoping to get through the transaction without any mission-critical misunderstandings. I think I will sound more casually fluent after the unrushed time I spent today, carefully attending to every subscript and superscript on every character in every formula for several hours.
The AIXI framework + it's self optimizing property
basics of aixi and relation to previous policy learning programs
Fernando’s precise formulation of the assumptions behind standard reward models.
Fernando’s lecture was extremely insightful and useful
Appreciation that the RL from yesterday can meaningfully be extended in the theory, and that studying preferences can tell us a bit about reward functions
AIXI and preference representations
Preferences under certain assumptions are generated by a utility function.
VnM theorem

Would improve6

I wish Francisco's two-hour lecture in the middle of the afternoon had been presented as optional. After carefully loading my head with all the unfamiliar notation from the morning, it was a shame to scramble it again. I had trouble staying awake.
material was interesting but didn't seem particularly relevant/important as a direction of study
The lecture from the second part, on preference axioms that "justify" RL, could have been more engaging. The material was still interesting though.
We didn’t really have time to work on Fernando’s exercises, that would have been very useful.
There wasn't really enough time on David's exercises to get far in, so a bit more time would have been nice to work beyond what was largely the technical setup, and I think Fernando's lecture could have been faster and with less audience interruption to convey this better
The exercises were too long.

Heuristic Arguments and Verification 1 run 26 responses, 17 written

for pasting into Claude

October 2026 Wed 28 Oct Aden Power 26 responses energy 3.9/5

Rated against what each person expected. 0 is as expected.

Overall +0.05 as expected

Content -0.05 as expected

Clarity +0.05 as expected

Motivation +0.05 as expected

Exercises 0.00 as expected

Readings +0.10 as expected

Other activities +0.13 as expected

Pace -0.23 about right

Difficulty 0.00 about right

Novelty +0.64 mostly new

Read what they wrote17

Worked well3

I suppose Q&A does not count for this feedback form -- but if it counts, it was really good and enlightening.
The talk was good and useful
The morning lecture was very helpful!

Would improve10

Make some of the lecture less vague
the morning talk was pretty ineffective. it was not really clear to me what was trying to be communicated beyond "we should be able to understand the outputs better than simply observing them". talking to Aden and reading helped me to see a bit better what is the ARC research project.
nothing
today was great, i wouldve liked some more intuition to start the reading, but that is a minor detail
I wish the ARC lecture was more explicit if possible.
Nice, maybe a bit more help with the actual arguments ARC uses would be helpful. Otherwise interesting!
The ARC talk didn't really have any concrete details and the presentation could have gone into the mathematics in more details. The rest of the day was mostly self reading.
The lecture in the morning on ARC overview and the exercise in the evening were useful. For the lecture I would consider including more real examples - say explaining in detail one of the examples that we later had in the exercise. Maybe this is not possible considering the time limitation. Overall there are no complaints.
Not entirely clear on the specifics of this research program.
Not much. I think the ARC research agenda is interesting and I was glad to learn more about it.

Changed my mind3

I really do not understand how ARC thinks that ambitious mec interp is going to work in general. Even very simple models are very very very hard to analyse.
Even though interpretability was not explicitly mentioned, the framing given by the ARC speaker makes me believe more than it would be useful. If we can use the interpretation of a part of the network to simplify part of the task of predicting the output, we are one step closer to understanding, and half a step closer to tuning/controlling for alignment.
Was glad to get an idea of what they were doing at ARC.

Other thoughts1

I think the 'worked better than i expected' style question is functionally useless given that you have no prior on my expectations. a simple 'how did you find x' on a 1-5 scale would probably be more informative en masse.

Instrumental Convergence 1 run 9 responses, 9 written

for pasting into Claude

June 2026 Mon 29 Jun Leon Lang 9 responses energy 3.3/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.8 of 5

Teaching and format 3.6 of 5

Difficulty match 2.8 of 5

Energy 3.3 of 5

Read what they wrote9

In their own words9

I found the content interesting! The mathematical formalisms provided a valuable way for me to think about concepts we'd only discussed in an imprecise way so far which was greatly appreciated. That said, I'm not sure I buy that this is quite the right language. It seems to me that any walker on state space with some sort of stochastic element will have a tendency to end up in denser areas. I think that the exercises could be made more challenging. I felt like I could solve many of them just by staring for a couple seconds.
- I liked the lecture - As Theo said, it would be nice to have a smoother transition from the exercise sheet to the papers, by introduction the correspondence between concepts in the exercises and concepts in the papers - I missed a more general discussion of arguments for power-seeking behaviour, separate from the idea of retargetability - I think the group discussions lacked good discussion prompts. We spent a lot of time discussing the detailed results of the papers in my group and it would have probably been more productive to discuss the general questions so that we could focus on the ideas behind the papers - The final discussion of limitations didn't work very well. Each objection might have required longer to discuss, and we ended up going on tangents, so we didn't have time to discuss many objections - I think a 3 min rant per person would have been better
I found the topic genuinely interesting although I think applications to realistic scenarios are rather limited at the moment. Nonetheless, the assumptions are spelled out in a clear way and the conclusions were accessible through small and motivated conceptual steps. Maybe the talk could benefit from a couple of slides to present some critiques of the topic. Also I'd try to adapt the notation in the slides to that of the paper(s). Overall, good module.
The topic was pretty interesting. I did not really feel much for it during the start of the day but it got better over the time and the learning outcomes I realized towards the end were definitely important.
It was an interesting topic, and one where a formalized approach has been attempted with some success. So there was something to learn and discuss, which makes it a good choice of topic. The presentation was ok, maybe it could have been a little clearer in its notation and statements. One idea for a discussion could be asking whether the definition of Power they propose is a good one, and how well it captures the idea we have.
Slides were good, but the text could have been larger and a bit more concise (which is true of almost any presentation). Exercises would have been better if solutions were separate or as clickable show/hide content (on a website, the agent foundations module had that nice). The readings were good.
I think the lecture in the morning could have been much faster if it was more organized--david quarrel's lectures were a great example of what today could be: state definitions, state results quickly and directly without belaboring setup or intuitions, and then set us free to look at notes, exercises, papers in detail. The perspective given in the notes seemed a bit more motivated than in the papers, but it would have been nice if there was some section of the notes that was done in the language and context of the papers to better set up the afternoon readings. more generally, I think the past four days or on agent foundations and adjacent stuff could be condensed a lot. I would have liked have more time to spend on SLT, computational mechanics, and mech interp, since I think there's a lot more detail to be unpacked there, whereas the recent days have just sort of drawn out formalizing philosophical ideas that more or less amount to choosing notation rather than being mathematically involved.
The presentation and exercise material were well-prepared and understandable. A bit too much time was allocated for the small-round discussions in the afternoon, I felt we ran out of discussion points at some point. The original papers use a slightly different formalism, and some transition would have helped to more quickly understand their equivalence.
author doubts his own proof.

Intro to ML Engineering 4 runs 107 responses, 198 written

for pasting into Claude

October 2026 Mon 5 Oct Dan Wilhelm 42 responses energy 3.7/5

Called “Intro to ML Engineering and Transformers” that cohort.

Rated against what each person expected. 0 is as expected.

Overall -0.05 as expected

Content +0.26 better than expected

Clarity +0.12 as expected

Motivation +0.38 better than expected

Exercises -0.02 as expected

Readings +0.04 as expected

Other activities +0.32 better than expected

Pace -0.31 too slow

Difficulty -0.52 too easy

Novelty -0.60 mostly review

Read what they wrote75

Worked well21

The live code examples were nice however they were hard to follow. I wish the code was accessible before the lecture.
The discussion was stripped down to the simple essentials, AND it involved live implementation. Both these contributed to efficient learning.
Long form disucssions.
Exercises and group work.
I thought the ability to ask questions mid talk was very useful for clarification.
the worked code examples
very happy to hear about the various opportunities (but i felt it was out of place at the beginning of "Intro to ML" class day)
The discussion during breaks.
I knew most of the material covered. Really enjoyed having the opportunity to discuss it with others who felt it was new. I also enjoyed watching the teacher code on the fly in front of others. That is super hard and I was impressed. (Helped me see what kind of skills an AI safety researcher should have).
The schedule structure is good.
The lecturer was very knowledgeable and answered all the questions well.
A very soft landing toward the basic but useful ML concepts was very helpful. The lecture was well organized and well explained. His answers to participants' questions from his own practical experience were very helpful to understand the general idea of the concepts. All the questions from the group were a good addition to the lecture.
mixing slides and theoretical explainations with exercises and implementations
its a good idea to give an introduction
Examples with TransformerLens and GPT-2
I loved the attention section, and I liked how easy it was to discuss questions during the lesson.
The lecturer is good and his answers were clear
I think the engagement with questions was nice.
Basic understanding of the transformer model structure
People were kind and receptive to chatting/working/eating together.
The exercise sessions were useful.

Would improve31

I think more breaks would be good. The second session was a marathon.
slightly more mathematical / formal clarity
The ordering of some of the concepts could be shuffled around to flow better for a math / physics audience.
More figures, and perhaps a better feel for the theory narrative: like constantly returning to central intuition for LLMs as a sequence predictor.
Some of the attention material was simply presented but not well motivated (in terms of where these blocks came from, why we made certain design decisions).
The explanation of the transformer architecture was a bit difficult to follow for me - structures sometimes seemed haphazard/unmotivated, and I got lost in understanding the various different parameters (especially when starting the coding exercises). But, it was not that the material was difficult per se.
Bit of a slow start
Some more diagrams and structure would be nice. Maybe a general overview of the transformer architechture to start. It may also be worth it to spend more time on the complexities of the transformers etc than on the very basic concepts at the beginning. Although there were lots of questions which makes it hard.
Maybe the presentation preparation could be better. (I found 3b1b explanation of attention much more intuitive -- but maybe he oversimplifies.)
it was not clear at all we would be using Jupyter notebooks. I specifically did not set my environment up bc the prereqs for this day were just "foundations" the material was very (too?) basic, bit the presentation was confused and unclear. the material should be presented more systematically by, for example, giving an overview of the architecture before explaining each part, defining the variables and where they live, given the intended academic audience.
Sorry for late feedback -- felt like some concepts could have been introduced more rigorously, given almost everyone is postgraduate level in maths, physics or CS. Useful review overall.
More breaks probably? I think also sectioning the talk into different parts would help better. So there's more granularity on what we're talking about.
There were many "why that choice" questions (ie, "please motivate that choice"), which I feel could have been better prepared in advance, knowing the sensibilities of a math-y group of participants. The lecture felt redundant with the 3B1B video series, or other tutorials. It may have been a good use of the day for the participants to be expected to be familiar with it in advance, and work in groups for discussions about the content, to solve various exercises, with some Colab guidance.
I think to get the point of residual stream across it may be helpful to show a counter example. Basically residual streams are added to by the (computations done by the) MLPs, attention heads etc. The counter example (what a residual is not) would be to show that the readout of one is the input of the other. I think this might help motivate the idea of the residual stream--why it is a "stream" where stuff gets taken out and added back in.
Not much. Since it was the first day, it wasn't technically very new, but I think that is expected.
Maybe It would be nice (I don't know if it will be done int he following days) to have more mathematical definitions of the objects we are working it (as example, in the transformers part, as generating kernel), in order to figure out what possible research directions would be
Better clarity for the slides and explanations. There are some typos and confusing explanations during the lectures.
I think the presentation could be improved. In particular, just practicing giving the lecture before and doing it less on the fly would allow us to spend more time on learning. I think visualizations would do great for learning how attention works, how the numbers flow through the vector, instead of solely relying on the mathematical equations. Perhaps even some interactive thing generated by claude code or something.
I could follow the discussion and it was well done. For some other participants maybe they could watch 3Blue1Brown videos.
I wish the speaker had rehearsed the presentation, or otherwise prepared it better. Though the speaker clearly knew what they were talking about in the big picture, he was often lost on details he "hadn't reviewed recently". Separately, I wish the presentation was presented with an eye to a mathematical perspective: the audience is mathematicians and theoretical physicists.
I would try make the room less stuffy. Maybe AC would help.
it was a little slow at the beginning and a little fast in the end. meaning I would have spent more time on the more difficult concepts and less on the easier.
the beginning on the "basics" of ML was a bit loose/not as clear the connection to more modern methods and some small mistakes, the attention part was pretty great but the programming problems are mostly bottlenecked by not rembering the syntax rather than not understanding the topic well enough, especially since the formulas are written on the sheet
I would prefer the coding challenges to be more difficult. Today they involved writing down math formulas in pytorch syntax, which is okay but does not teach me much.
More breaks. I found it hard to focus with such a long session.
The introduction could have been more energetic and engaging.
Some references for mathematical and computational precision.
Some typos, nothing serious.
Make the lectures slightly faster
I think more examples would be good.
Didn’t have much expectations for an intro/refresher day but always happy to recall stuff!

Changed my mind19

No changes.
not really, but we were only covering an intro to ML
Just how much LLM building is trial and error
considering applying to this funding! meta: is this the level of preparedness / knowledge of successful researchers / funders?
I had a time to properly think about transformer architecture, which was useful.
Seeing Dan work in Colab was instructive!
I had a really interesting conversation at the beginning of the ice-breaker sessions. I learned some ideas about decision theory that I had questions about. It also opened up new questions that I plan to follow up on.
Didn't change my mind on anything today, just a good review on transformers.
Before the program started, I was worried about the level of mathematical concepts. This first day provided a good overview of the practical and theoretical understanding of AI/ML models. I got much more comfortable participating into the program.
I was surprised at just how simple the recipe for a transformer is.
On prospects of procuring required resources from notoriously obscure sources.
today was mostly notions, nothing really worth mentioning. I definitely learned a lot!
not much this was more introduction
About possibly applying for a founder's programme.
No
n/a
No change.
We could access the modalities(parameters) of any model
I feel more confident in my ability to pivot to AI safety given how much more at place I feel here than expected.

Other thoughts4

if the material were less familiar, I would expect to have gotten less out of it, as some of the presentation was not (sufficiently) self-contained
Thanks for bringing us together.
Nope
no

September 2026 Mon 7 Sep Julian Schulz 41 responses energy 3.9/5

Called “ML Engineering” that cohort.

Rated against what each person expected. 0 is as expected.

Overall +0.05 as expected

Content -0.02 as expected

Clarity -0.07 as expected

Motivation +0.02 as expected

Exercises -0.29 below expected

Readings 0.00 as expected

Other activities -0.05 as expected

Pace +0.17 about right

Difficulty -0.32 too easy

Novelty -0.29 mostly review

Read what they wrote77

Worked well27

I enjoyed the lessons before the last hour acceleration.
I really enjoyed Julians talk!
thinking through the design of the adam optimizer together was fun
I liked that the teaching was interspersed with exercises
It was nice to have a bit of time at the beginning of today to meet the other participants.
meeting the other participants and lunch was good, i think this was roughly the main purpose today?
Talking to people was useful and getting to know everyone.
Socializing was great.
The introductions were very nice, I enjoyed chatting with a series of participants one-on-one.
The hands on exercises and the capstone project.
I appreciated block-diagrams and examples. It would have been useful to introduce notation in the presentation deck, instead of having to ask and receive that information orally.
Intro to ML lecture was delivered very well.
Discussions to get to know people.
The networking with prompts
Lecture split with exercise sessions allowed to not get tired thanks to context switching. Also the breaks. The slides were good as well, albeit not sufficient for proper explanations, but that's perhaps too harsh for the overview.
- claude capstone project was very enjoyable, best part of the day - socialising/ ice breakers were effective, met nice people and didn't feel awkward/ intense as some forced ice breakers can be
Good and quick recap of things to get us warmed up
Stopping frequently during lectures for exercises and relaxation.
The short interactive parts are good, both to break up the flow, but also to get people thinking on their own, which always helps learning. The conversations during the first half of the morning were very good for getting to know people. Different formats could be beneficial to have more varied interactions (i.e. discussions in smaller groups (of 3/4/5 ppl)), and these might be better matches for some of the questions/prompts given, but these would let you get to know more people faster (which becomes more important the bigger your cohorts become).
I enjoyed the get-to-know each other in the beginning. I really enjoyed the ML fundamentals lecture, I think it was very well prepared and understandable and engaging.
The questions from the audience and answers from the lecturer were great.
Open questions to class, the Claude sessions
It was good that we spent some time introducing ourselves to each other. I gained a lot more clarity on the background of other participants, and their motivations for the intensive/fellowship.
Having a small group to learn the basics of Claude Code
I liked the exercises format. Both the claude code ones and the notebook ones.
The claude code exercises were good, although not described a priori very well.
I thought Adam explained a lot of the features of Claude Code and coding agents really well, and I got a lot of clarity on a few things I didn’t know as much about initially.

Would improve31

Better explanations of technical concepts (such as transformer architecture), and I would have enjoyed a deeper (perhaps more formal) approach when it comes to teaching in general. I think that there was not a great time control. No time for solving the exercises, and I think the socializing exercises took too much time.
Sometimes when the group was split, it was unclear what the options were regarding the possible choices. Discussion games at the start could've been improved a bit, but I think the main restriction was the room shape. Would've been nice to mix a bit more, and maybe having another in the future where you talk to other people would be useful.
maybe scale down the introduction part in the morning and split the intro to ML into 2 sessions, by the end of the ML intro I couldn't focus on the content which was still very interesting just due
Adam was great, but I don't think the "claude basics" activity was super useful. I felt like it was an intro to vide coding, but I think it would've been useful to go from the very start to the paper replication and getting hints as to how this could be done efficiently. Learning about worktrees whilst doing this project would've been more valuable.
I found the today's content very helpful but I felt like it went too fast. I think I would've appreciated if the same content was instead spread over two days. As someone who had a low level understanding of about half the content before hand I would've liked a bit more of a chance to work through exercises and have the instructors roaming around for questions.
I think choosing between ML fundamentals + Claude Code intro is not the most ideal - naturally I wanted to cover everything so chose the former which left less time for the latter. I wonder if the first half of the day (intro stuff) could be shortened, or start an hour earlier, to fit in both?
We were given a couple of choices today for what material we could learn. I think it would have been helpful to know in advance what was going to be covered, as there was a bit of a rush to decide (and without really know what would have been done in each session). I did the ML basics in the afternoon, but in hindsight, I think intro to Claude would have been more useful for me. My answers below reflect the fact that I did the ML basics, parts of which felt fairly familiar. I think it might just be difficult for people to choose, so I'm wondering if it'd be possible to compress the ML basics / intro to Claude so everyone gets to do both? I felt the "try to get Claude to replicate the paper" exercise was a bit less helpful. Perhaps it could have been helpful to have a bit more guidance on this part (what would be the "best practices" way of doing this)?
If you put formulas on slides explain the terms (what is the sum over, what is y and what is y hat). I would prefer to do deeper reading over a rather superficial lecture. Smaller group size would help for that.
The Reinforcement learning section was somewhat too quick, I would have spent some more time on that.
It felt that the ML/AI intro was not too well structured. Arguably, that's a common problem for many such intro classes. E.g. RL was surely too math-y compared to the beginning, and perhaps not as relevant in fact to RLHF (or the model lifecycle is not relevant to "intro to ML"). I also felt that there is a real discrepancy in participants' knowledge (it was to fast and superficial for those who didn't know it before; boring for those who already knew), so it seems reasonable to run a small test in the morning of the first day and then split into groups where the intro would be more targeted. Also, it looked like the diagrams in the presentation were a bit rough (in my experience it was similar to unpolished tikz code generated by a model). Finally, it may be worthy to emphasize that this is not academic setting anymore, so people should not be afraid to give wrong answers -- at times there was a bit of awkward silence. Socializing part was cool, but the room is too small for 40 people talking simultaneously, so it was overwhelming.
I'm not sure if I got very much out of the Claude Code session. I learned a little more about how it works, and a few handy tips, but nothing that I expect to really transform the way I use it. This may be my fault for choosing said session; I already use Claude Code quite a lot and was at least somewhat aware of most of the features we covered. So my only pedagogical suggestion is probably the session could've been a bit shorter. Working through the exercises was mostly a matter of waiting while Claude ran, only a little bit of the time was spent trying new things.
Claude code exercises are poorly explained and assume a lot of prior experience using claude code.
The session in the morning designed to get to know each other was way too long in my opinion. After the session, there was also a break and lunch which achieve the same objective. Because we spent all morning on this, the afternoon session was a bit rushed. I would try to include a part of the afternoon session into the morning.
It's better for Claude Code exercises to have more structure and well-defined goals to verify your abilities upon. Another way to approach it is to build it around designing skills and workflows yourself. Yet an exercise should incentivize the participants to do it.
Voice projection. Unfortunately, it was very difficult to hear instructions when organising the introduction rounds, or to understand when there were breaks. Regarding the lecture: a poll among participants well before the course would have saved awkward hand-raising moments to decide what to cover, or how to distribute groups and in which rooms.
I would not do the 'intro to claude code' vs 'intro to ML' split - I think it would be better if everyone does the same thing so it doesn't feel like you are missing out on an option.
Pace could be a bit slower. Temperature in the room is too low.
Improve the pedagogical value of the Claude Code exercises. I didn't feel like I knew what I was supposed to do. Most of it was guess work and hoping things would work. The paper replication is the same. Some guidance would have been interesting instead of simply saying "reproduce figure 3 in this paper". The use of the research project template itself was not well introduced. Slow it down, narrow the scope and pass a cleaner message.
While I appreciate that this intensive needs to cover a lot of (rapidly evolving content), I think the speed at which the fundamentals were covered clearly divides our cohort into those who are more theoretically-oriented with almost no AI or coding experience and those that have used AI tools.
Define the variables in all equations or include on slides
I think I was missing a clearer vision of what will be taught in the fours weeks, this week, and today in particular, how this fits within Iliads vision, what we want to take away from this etc. With the afternoon exercises on agents, I think I would have benefited from a sort of best/good practice for agentic coding. the exercises were nice to get us working straight away, but it was often only through questions or confusions that better workflows became clear to me
Desk placements (happy to help with moving them around!) into more of a school / uni structure (facing the lecturer). Clearer outline of what we would be missing out on if choosing to attend one session rather than the other / choose one exercise rather than the other. Ideally to announce such choices beforehand so we can say skim over the materials and decide in a more improved manner [Generated by my FOMO]. Also the air conditioning over the tables closer to the door make it chilly, and not just if you are under it -- I would switch it off for the afternoon sessions.
- would be nicer to have lecture notes, especially ones that are clear. since this is an earlier program, it makes sense this doesn't exist. suggestion: look for volunteer scribes? you can ask for 2 volunteers per day and get claude to smush it together (although i would put high emphasis on creating lecture notes which don't read as ai generated, this is very off putting) - similarly, it would be nice to sort of do a pass over any slides / material. i think using models to create materials is efficient and fine, but one should do a pass over it or use it to create a skeleton as reading something that reads as "claudish" can be similarly off putting
Introduce motivation of something with pictures during the slides. Sometimes it was not clear. Rest was great. Especially the discussions during the lecture and the exercises.
Simply too little time. I think it's worth extending to get everyone up to speed. This naturally depends on the coming days and in what way they rely on deeper understanding of today's material. Also, the people who most need a break are (generally speaking) also the people who would most benefit from the excercises, i.e. they are the people with the least prior experience. Please keep this in mind when offering for people to solve the excercises during the breaks.
We were given a choice between different exercises/different lectures (either ML fundamentals or Claude Code, and then either Claude Code capstone project or exercises). I found it difficult to choose. I think I didn't have enough information to make an informed decision. It would have been useful to know more precisely what each of the options would entail beforehand!
The introductions parts were working fine without suggestions on what to talk about with each other. Otherwise everything perfect.
Perhaps a more targeted course (i.e. covering less topics), and have people read the prereq. material one week earlier! The Claude exercises were fun, but there were some small API topup issues
More time for exercises. But maybe this will improve in the coming days, because this was just the first day.
The lecture for the ML basics lecture was too rapid for me. I understood what I already knew, and not the concepts I didn't know. Maybe less time for people to try and come up with various gradient descent algorithms from scratch. There are lots of reasonable answers, hard to converge on momentum etc.
I worked through the CC exercises with Adam. While I appreciate that the exercises were a substrate for learning about CC, it felt a bit strange to work through knowing that frontier models can probably one-shot these exercises with no need for e.g. subagents. It might have been cool for the exercises to really show the power of these techniques if possible.

Changed my mind14

Being the first day, got to know other points of view about AI safety, from other participants. Discovered other AI safety institutions and interesting career paths.
n/a
I was expecting today to learn in depth the transformer architecture. Instead we spent a lot of time on basics. I also expected to work on the exercises and set up claude code with vs code before but there were other necessary setups eg GitHub that we were not informed of in advance. Overall I would have learnt more by working through the transformer material on my own and spent rest of the day on the exercises.
n/a
Claude Code in VSCode is a bit clunky
No
1. Gradient descent was improved heuristically rather than mathematically (Newton's method) or physically (energy-based approaches which give smth similar to Newton). 2. That is sometimes does happen that an input from the engineer leads to an improvement (RoPE giving better results than basic encoding) -- which is not how it usually works.
n/a
Not yet
Yesterday night I was feeling very anxious and feeling like an impostor, and I thought the atmosphere at Iliad would be much more competitive, I was happy to find out that I needn't have worried about that!
Attention as a mechanism is more complex than I thought.
Not really yet!
I should get familiar with coding agents
N/A

Other thoughts5

no
n/a
1. Do I need to say in advance if I need to take a call at 17:00? Can I book the box in the Iliad building ? 2. I cannot attend Friday 11th and need to leave at 17:45 on Thursday.
Ad question on exercises: I only really did one exercise (pair programming with someone) - implementing the SGD update. I thought that was pretty easy but I actually learned that you cannot do self.params = self.params ... so that little exercise was very helpful!
I will have a doctor’s appointment on the 14th to forewarn I may not be able to attend in the morning.

June 2026 Thu 18 Jun Julian Schulz 12 responses energy 3.7/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.6 of 5

Teaching and format 4.2 of 5

Difficulty match 3.3 of 5

Energy 3.7 of 5

Read what they wrote12

In their own words11

10/10. No comments.
The lectures were very good but I was lacking background setup and experience to engage with the practical exercises. I focused on theory instead, read some more literature on transformers.
It was very nice, both the teachers were very helpful. I hope we had one more day of this.
What worked well was the abstraction first framing and the immediate actionability. I turned the redundancy condition into an SAE experiment this week and found it predicts cross model feature universality, which made the day especially useful. A short worked example for the condensation correspondence bound would help the second paper land more cleanly.
- The hands-on agents exercises in the afternoon were great - I think it would have been useful to have a clearer introduction at the beginning of the afternoon - In the morning I participated in the advanced session, and setting the tools up was very useful - A demo would have been nice, I feel like I only had time to set things up, but not to test them much
I think I shouldn't have taken the beginner route in the morning. I was inexperienced with pytorch and i wanted an intro to that but i was familiar with transformer architecture stuff already so didnt need review on that. in the afternoon, I spent all the time on setup, since using an api key for claude code on windows made things really difficult, and the recommended setup was impossible for windows.
I would have preferred more time for exercises in the morning session ("beginner room").
very nice to be able to have a taste of how easy it can be to set up things on runpod, and using claude code in multiple terminals, I'll definitely get the higher usage limit for Claude now.
The morning session on basics of ML architecture was fine. From the afternoon I got that Claude Code could be very useful if one wants to code - I just have no sense for what it was doing.
Content was great. It was nice to see many best practices from people doing ML research every day. Some of the setup was much higher friction than I expected, possibly it's great to ask some Iliad colleagues to install and test beforehand.
The morning session was great and I learned many new technical tips. The second was also good and I discovered Claude features that I wasn't aware of. However, the teaching format can be improved, maybe starting with some examples demonstrated by the tutor before moving to the exercise of reproducing figures of a paper.

Other thoughts1

Strong day. Rare for a maths heavy session to translate into a runnable experiment within the week.

April 2026 Sun 5 Apr Julian Schulz 12 responses energy 3.3/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.1 of 5

Teaching and format 3.7 of 5

Difficulty match 3.0 of 5

Energy 3.3 of 5

Read what they wrote34

Worked well11

Just learned a lot, tinkering yourself with the notebooks was very useful
I liked the applied section (track B) and then the lecture after with tutorial.
Self-selection into tracks makes a lot of sense for this topic, given the diversity of backgrounds.
The track B tools we learned about and implemented will be really useful I think.
Closely scaffolded coding tutorials
Format and teachers were good. Track B content was also useful to see, as well as some of the RL content at the end
Breaking up into multiple tracks depending on your level was good. I was in Track B. A mix of practical-work-through-these-exercises and then having a lecture in the afternoon was good for energy management.
Julian’s energy was infectious. The community did not seem competitive or shame-driven.
Letting people in Track B work on the instruction individually > pair programming
For Track A (Julian): great combination of lectures and notebook exercises.
Exercises were good, pair programming was good

What they took away12

So much! Just generally going over all these structures, from the bottom up, was extemely useful
Lecture part on reasoning models was really useful and also the tutorial of implementing an attention layer.
Transformer architecture
How to work with Runpod probably, and a quick overview of RL
Probably runpod
familiarity with the ease of vast ai
I liked the very end about RLVR
How LLMs' chain of thought metholody are trained on verifiable outputs such as maths and coding.
I got lots of links: I think/hope more learning will happen later, when I have more unrushed time to go through them all, than actually happened on the day.
remote SSH configuration with vast.ai; supervised finetuning and RL.
How RLHF works, and how to train reasoning models - at least the beginning of this.
Reminder of some practical details and implementation details for transformers

Would improve9

Smear this over yesterday and today!
Understandably the acquisition of a GPU cost a lot of time but that would be great to fix. I would have had enough time to finish track B’s content otherwise, which would have been ideal.
Better setup and exercises for track B. Lunch was too spicy, I'm sorry 😭
The logistics of the runpod credits
I would have liked to see a coding session, especially if it's an experienced software engineer showing how they use an agent to get to a solution and work through a problem. In Track A there was a lot of conversation about agents at the end and how useful they are even for software engineers (per the TA of that track).
I would have liked to get the document with the links describing the content of the day, before the morning of.
1. More assisted individual work: I found having more TA accessible rather than just relying on 1 instructor is useful. Less pair programming: pair programming makes it more high-cortisol and challenging. 2. The pace of afternoon session - Lecture 2: Julian spoke too fast, rush too much, I would also swap slides with interactive website (Claude can make one) to explain the structure of transformer/attention rather than static images/slides. He links to a LLM visualization website, which is a good source; but doesn't replace having an interactive website to visualize the architecture itself > slides. During the lecture he also asked people to raise their hand if people know/don't know a material: this honestly wouldn't be accurate since social pressures would drive people who don't know a concept to not raise their hand. It's much, much better if these lectures can be released 2 days prior: how it works can be: you make a form that asks key questions over the day's materials: from 0 to 5 how confident are you with Concept A, B, C => At the end of the survey you share the link to the day's content document. This would make more people to fill out the survey and more attentive to their own weaknesses, and fill out any knowledge hole before the lecture; furthermore, it's better for people to self-learn/review concepts and come with better questions/better concentration. 3. The 1.B was good, but configuring remote SSH took an entire morning and was very inefficient. Could this have been prevented?
Hope to learn more about RLHF and get some hands-on experience?
More time on transformer pair programming, slower presentation of details around LLM training (as I found the content on RLHF and RL largely incomprehensible where the setup details were not so clear from the slides - fully written notes would have been a lot clearer to me)

Other thoughts2

Would love to be able to access content earlier, so I can prep the day before. And also the days I cannot be there, would love to be able to still ask questions in discord/review the work early.
We should check out more Vietnamese or Chinese restaurant please.

Launch Day 4 runs 62 responses, 82 written

for pasting into Claude

September 2026 Fri 2 Oct Leonard Bereska 20 responses energy 3.8/5

Called “Launch Day + proposal round 2” that cohort.

Rated against what each person expected. 0 is as expected.

Overall +0.11 as expected

Content 0.00 as expected

Clarity +0.17 as expected

Motivation +0.39 better than expected

Exercises +1.25 better than expected

Readings +0.40 better than expected

Other activities +0.63 better than expected

Pace +0.44 too fast

Difficulty +0.28 too hard

Novelty +0.17 an even mix

Read what they wrote28

Worked well7

I think I ended up with quite a good proposal :)
Proposal days are great!
Direct feedback from mentors on proposal feasibility.
THe fact that the intermediate submission coincides with the final one. Also, the Launch Day presentation! Very helpful
Getting feedback from other people
Kyriakos (I believe that was his name) is fantastic and really helped me out with his advice on my proposal
Networking with the mentors / fellows.

Would improve11

It's a little difficult to feel like there's enough time to come up with a project proposal, but it's nevertheless a fun and rewarding exercise.
More 1-on-1 mentor matching slots.
I don't know for sure -- but developing ideas for theoretical projects, especially on Agent Foundations, was definitely harder in comparison to the empirical ones. I wonder if it's better to ask people to start thinking about the proposal, and announce that it will share the identical structure to the first one several days ahead.
Not sure proposal days are valuable for me.
I think the same as from previous proposal day
More structure please.
I thought this was great.
I enjoyed today.
Nothing had a great day! Actually, I think it was a bit annoying to work in groups during the last proposal day and then alone today, I felt I would have rather inverted it, or had both days the same, or had the option on both days
In the last project day, we were strongly encouraged to work in group. Today, instead, we were asked to work alone with no further explanation. That felt a bit arbitrary.
I think it is really difficult to come up with a proposal in our situation

Changed my mind6

Reading more about debate, I think it is a very interesting topic and something I want to look into more.
Sharper scoping makes high-risk research questions tractable.
Coming up with a novel proposal is hard when Claude tells you that it has largely been done. Very discouraging. But it is the research reality, and it's good that literature review has become so much more simpler.
n/a
I feel like on the last day I have become very driven to do something, I dont really know what just yet and need to figure that out, but I plan on working on some projects over the next few months
I got some advice from Zach on my future career in AI safety which I considered really helpful

Other thoughts4

Incredible program overall, huge thanks to the team.
I have slipped on feedback forms starting Tuesday this week, and only filled them today. I am wondering how common of a phenomenon that is. Sorry for that nonetheless.
n/a
Please change the feedback form so that it matches a day almost entirely devoted to drafting a project. Thanks!

August 2026 Mon 24 Aug Leon Lang and David Udell 21 responses energy 3.9/5

Rated against what each person expected. 0 is as expected.

Overall +0.38 better than expected

Content +0.33 better than expected

Clarity +0.10 as expected

Motivation +0.43 better than expected

Readings 0.00 as expected

Other activities +0.67 better than expected

Pace +0.10 about right

Difficulty +0.05 about right

Novelty +0.62 mostly new

Read what they wrote15

Worked well1

Both the quick overview of programs & the presentations were very helpful (and moving the presentations earlier in the week was a very good idea, there are several people I still want to chat with about their ideas)

Would improve11

Just make launch day better, or give up on it. 4min was too few, I don't even know how it is possible to form some impression about people in 4min presentation aside from metrics like "good at public speaking". One idea would be just include the whole day to presentations and then more time for everyone. Or make people record presentations of at most 10min in a website and send to you guys, so you guys can see (if the idea is to get points for the fellowship). In general this day was somewhat communicated poorly, I don't know. Content day would've been way better. The morning AIS overview of funding is perhaps helpful for newcomers but I bet most people here were familiar with eveything there (and if not, it's on them). For Ryan Greenblatt's text, it's just too shallow, if it is to do something with the content of the text, just give us more time and we read AI2040 and discuss properly (although I also think a theoretical day would be better).
I don’t think I learned much from the presentations, I would consider dropping it.
4 minutes was harsh for talks.
The whole presentation idea was a very nice breath of fresh air from the usual technical content we go through. It sort of put taking part in the intensive in perspective and made us actually think about immediate future steps like projects grant applications etc.
I would move the presentations for the last day of the Intensive.
I think the primary thing was that I felt that the presentations could've been organized better. I think initially on Discord we were told that we'd have 10 min to give a talk, but this was cut to 4 min day of because more people than anticipated wanted to talk. Perhaps if there was a poll or something on Discord then this could've been better handled.
Not sure if this is a feedback for the morning session, afternoon session or today as a whole. David's session was pretty useful for people who are newer to the field which might be most of the participants. Morning session with Leon was fine, I would include the AI Control reading material from BlueDot Impact course since it describes the agenda in a way that would capture broader audiences. As for the afternoon, there were a lot of interesting ideas and overall they are useful talks. I appreciate that Zach paid attentions to the participants and ask a lot of good questions. I also wish other teachers were doing something similar.
The reading was near useless - too scant on details (but it is also bad to be too long). Today was very uncoordinated - most students didn't return on time after the break (no gong was hit) delaying things a lot. The discussion was weak (small group discussions would've been better). The launch day itself was disorganized and the main lead on organizing seemed way too eager to just cut people off from presenting. We should've had more time each.
I think the problem today was that because I got much less time than expected (I expected 5-10 minutes) and everything felt rushed, I was taken a bit aback and the quality of my presentation suffered tremendously (like, I felt I didn't have time to put the presentation on my screen as well as the big screen, which meant that I lost my anchor, and I felt I had to rush, and then it felt like almost no meaningful words came out of my mouth). To improve this: Next time, figure out a few days before who wants to present, assign slots, and ideally, have all the presentations on one computer (as in, have people send them to you the day before).
It would be nice to schedule some time to work on the presentations. Now I feel like they were quite rushed.
The ordering system

Changed my mind3

Over the weekend I decided to commit to AI safety.
There are many directions to proceed
Learned & thought about some research directions (some feasible generaltions of DLNs and what is known about their training dynamics etc.), thought more about career planning

June 2026 Fri 3 Jul Leon Lang 9 responses energy 3.8/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.3 of 5

Teaching and format 4.4 of 5

Difficulty match 2.2 of 5

Energy 3.8 of 5

Read what they wrote7

In their own words6

Good way to end the Intensive. Fernando's session was good and David's talk in the morning was laced with all kinds of insider-info which I didn't really know about.
The paper about misalignment risk categories was not too convincing to me, I thought it contained too many guessed numbers (how long until research pays off, how quickly would AI take over) and had too strong assumptions (we'll be able to contain slightly super-human AI safely to do research for us) and missed risks other than AI takeover altogether (e.g. gradual disempowerment). The following discussion, however, was interesting and informative - arguably more so than the article itself. I like the goal of ending up with a project idea in general, but the time for finding a project was too short. I think it would have been much better if this goal had been clarified early on, on the first day, so that one has more time to think about different project pitches.
thinking about timelines and different plans was very helpful for thinking through my own priorities, making a project plan was extremely valuable, but should have been given more time, maybe even much more time, as in like indicate this throughout the intensive already. similar for the opportunities after, it's unclear how naturally the fellowship follows from the intensive and what information is used for evaluating candidates apart from vibes, could make process a bit more clear (and maybe use for motivation during the course)
I think it was a good wrap
Nice way to conclude the programme; the session with Fernando was very good and stimulating.
- I think it is a very useful exercise to come up with a research idea, even if it is not a well thought out research idea - I would have preferred to be told with some time in advance that today we would present project ideas. If so, I would have spent a bit of time coming up with a project idea. And I thought that would have been very useful. - The discussion today on plans A,B,C,D was useful - I feel like "the usefulness of theoretical/foundational research" has been an elephant in the room from the very beginning, and we should have addressed this before

Other thoughts1

thank you for everything. i came in as a builder and now i am leaving with a clear research direction, a real first result, and the confidence to keep going.

April 2026 Fri 1 May Leon Lang 12 responses energy 3.8/5

Called “AI Alignment: The Field” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.2 of 5

Teaching and format 4.2 of 5

Difficulty match 2.6 of 5

Energy 3.8 of 5

Read what they wrote32

Worked well12

I liked the big overview of the field, it's rare to have that. Also the structure with the projects in the afternoon was good.
I enjoyed the opportunity to present and be responsible for my thoughts and work, it wasn't quite as much so with any earlier task, save for Stephan's math exercises maybe. It was the case that even if I may have done a bad job at it, I found the process valuable.
I like the Greenblatt article, although I had read it before
1. Q&A of project idea
Prompting people to think about project ideas was useful, as it motivates more active kind of thinking (as opposed to passively taking in information).
The idea of coming up with research proposals and presenting them was a good idea. I also like the exercise of writing down next steps that are feasible and that we will actually act upon.
Yeah I mean, career options were lovely, weren't they?
The project brainstorming was a useful exercise, and it would perhaps be useful to have more versions of this spread throughout the course (or more warning/prep time)
Informative, I like the ideation
I ended up speaking (usefully) to [participant] for much of the afternoon, so not much comment - the morning was good, in both the reading and the session on organisation layout
I liked the common brainstorming session where people presented their project ideas and Fernando + the other participants came up with comments & questions.
I just think it was super useful actually

What they took away12

The overview of the field.
The clustering, visions and ethics of potential programs, fellowships and employers was new and useful. I would have appreciated more detail, especially high in confidence, about them, but it was helpful as it was.
Differences between different grants
1. best way to prepare for coding tests 2. breakdown of funding priority 3. overview of all jobs & training landscapes, very useful.
The project idea I thought about I suppose, although it could use a lot more refinement.
David's overview of the various different jobs/fellowships/organisations was extremely useful.
aisafety.com
All of the job org conversations (esp XORLabs, Principia, Stormglass, which I wasn't very aware of) was very useful
All the different oppurtunities available
Org layout
That the Astra Fellowship is "as good as" MATS (according to David).
Hearing everyones ideas is just super great

Would improve8

I would let the conversation about the overview of the field and its institutions go on for longer. It's nice to get insight on what people who have experience in them share their knowledge, I learned a lot from it.
I would still appreciate if we received more concrete opinions, pieces of knowledge, directives that help narrow down the field and set of places to apply. (This covers my above take as well.) And probably this can only be effective beyond a point if each participant (or a handful) gets bespoke advice, which may take too long or need more TA's but I think it's worthwhile. (A counterargument could be that we had access to that during the entire course, since the TA's were friendly and available for discussion, but still making the process more structured helps.)
I think Iliad strawmans control pretty regularly
The overview of organizations at the start could have been compiled into a document / slide(s), which would have made it a bit clearer and easier to access as a reference.
I think the discussing about this plan A, B, C, D got stuck in a not very interesting/useful way and it would have been good to move on from that point to other discussion points before running out of time.
Uhhhh instead of drawing the career advice on the whiteboard, why not write/edit it in a Discord message?
More time for the brainstorming would have been useful, or maybe some small group workshopping before the big discussion could have also been useful
Maybe a more structured presentation of the overview of the funding and jobs landscape

Mechanistic Interpretability 5 runs 133 responses, 228 written

for pasting into Claude

October 2026 Wed 7 Oct Dan Wilhelm 38 responses energy 3.9/5

Rated against what each person expected. 0 is as expected.

Overall +0.45 better than expected

Content +0.66 better than expected

Clarity -0.08 as expected

Motivation +0.61 better than expected

Exercises +0.03 as expected

Readings +0.15 as expected

Other activities +0.40 better than expected

Pace +0.74 too fast

Difficulty +0.50 too hard

Novelty +0.79 mostly new

Read what they wrote64

Worked well20

The second instructor was very well versed in the material and engaged with everyones questions.
very interactive. Participants actively engaged with lecturer by asking good questions and we discussed freely.
The overview was broad and looked comprehensive.
I liked the coding exercises to reinforce the concepts we learned
Exercises
The brief presentation given by one of the students. Involvement is nice and sharpens the attention.
I think the afternoon lesson was quite nice -- due to examples.
The first half was well taught and quite clear.
Helped me get an initial but a very useful broad overview of different mech interpret techniques. Starting with CNN helped understand the history of such methods. Later, although there were lots of different ideas presented I understood and am taking home some useful stuffs.
I think the exercise where we had to deduce the type of circuit from the attention pattern worked quite well to help me internalize the method.
The afternoon session was excellent: good pace, good explanation, good examples & slides.
Lots of discussion
Julian Schulz' presentation was _amazing_. clarity, enthusiasm, knowledge, humour. overall great. the exercise with the K- and Q-composition was brutal (especially because the first day's on transformers was not clear enough to learn the architecture–i'm glad i spent the evenings learning abt it) but enlightening
The circuits expansion was particularly good.
Discussion was great during the lectures, i wish we had had more time.
I really enjoyed the balance between Dan's transparency and pragmatism on the basics of mech interp and Julian's enthusiasm on the newer methods/approaches.
I particularly found the live coding demo useful for the activation diffing.
breadth of the material was good, but too much to follow, like i should know all of this but maybe in more depth but I understand its the intensive
Discussion was helpful for having people break down concepts and ask questions rapidly
I thought the questions and answers were v helpful

Would improve25

The pace was faster in the afternoon than in the morning. This might have been due to the teachers' individual preferences. However, I would suggest maintaining a more consistent pace throughout the day.
I would add more specific examples into these slides--I feel like I best understand what's going on when there's specific examples. (This is in part because of my background: I am a mathematician and feel like I understand the ML setup less well than the theoretical one. This might not be true for everyone.)
Talk slower
The pace was quite fast in both halves. Splitting this or pre-readings would have been much more helpful.
The speaker talked too fast and the explanations were confusing. I had to read the cited papers to understand what was going on.
More structure, fewer slides. (E.g. slides don't even define the J-space. It was defined verbally in 5 seconds, and, as I understood after asking an AI -- not even correctly.) I think if we do exercises at all, we could discuss the details of these python libraries. Also too many vague questions from the audience, very hard to keep the focus
A little bit more discussion and hands-on, and more intuitive examples (metaphors) to help understand the complex new concepts.
I would increase information density
The part on CLTs was difficult to follow for me.
Explanations could have been a bit more clear.
very quick explainations of many difficult concepts
Great! Honestly a lot of information, maybe a bit slower through the more recent mech interp stuff.
I really enjoyed today. We did take a lot of time to go through older methods and I would've liked to spend more time going through more recent stuff, especially J-space. That being said, I had pretty decent exposure to previous interp research before NLAs and J-space.
More time for the Mech Interp session, part 2. That, or references to expository materials.
Was excellent
Better explanation of what circuits are. May be give more concrete examples.
again, i found the Dan Wilhelm's presentation extremely underwhelming, way below the level or rigour or technical knowledge needed for this audience
The slides could be more precise.
I wish the style of the presentation was more mathematical, even if that would require a little more notational baggage. (e.g. https://transformer-circuits.pub/2021/framework/index.html )
We probably needed the full day to get through the material, so more time would be great. Questions and discussion were helpful.
Not much, really enjoyed today!
maybe a little more time for exercises, but there was a lot of material so I guess it is hard to fit everything in.
less material more in depth
Less time on introducing Mech Interp, more time on deep-dive and math/matrix multiplications behind concepts introduced
More equations. At least for me, I find the graphs by themselves hard to interpret without the relevant maths alongside.

Changed my mind16

Random trial ams error seems much more effective than I would have expected.
I did not mech interp was used for safety and alignment.
I thought mech interp was doing pretty poorly but it seems like Jacobian lenses are worth more of my time.
N/A\
J-Space is interesting!
Not in particular.
Circuits and NLAs are, perhaps, more impactful than I had thought.
Mech interp is cool
not really, i guess i was surprised to hear that the more "mechanistic" technique of mech-interp are
`The mathematics behind mechanistic interpretability is close to non-existence.
Yeah, I now find these methods more interesting.
Feel more comfortable discussing research and collaborating w/ people here in the near future Feel more motivated to start diving deeper into mech interp
not really, but developed my understanding of circuits and cross encoders
the NLAs were really cool, the fact that LLMs had hidden logics that NLAs can detect was pretty cool and new
I learned about J-space for the first time today and I will definitely look into it more. I also feel that my impression that SAEs were a dead end due to the Deepmind deprioritization was overblown.
I didnt realise how quickly mech interp was changing as a field

Other thoughts3

N/A
Bit more of exercise, like doing coding would have been better. While coding, we could understand better.
n/a

September 2026 Tue 8 Sep Julian Schulz 41 responses energy 3.7/5

Rated against what each person expected. 0 is as expected.

Overall +0.27 better than expected

Content +0.27 better than expected

Clarity -0.27 below expected

Motivation +0.27 better than expected

Exercises -0.15 as expected

Readings -0.25 below expected

Other activities -0.20 as expected

Pace +0.68 too fast

Difficulty +0.54 too hard

Novelty +0.63 mostly new

Read what they wrote78

Worked well22

I think the lecture was great, I liked that there were questions to the audience, and I also love the questions from the audience, they are often very thought-provoking. I think the lecturer does a great job answering questions and I also respect his epistemic honesty. And he is enthusiastic which makes it easier to keep paying attention
discussion with peers helped clarify concepts
I really enjoyed the overview of Mechinterp - It was really helpful for me to get a good overview of the area so I can go and learn more about what i'm interested in.
I think the explanations of the topics, and then going on to neuronpedia to actually see it in real models was helpful. I also thought that the exercise of thinking about the induction circuits was helpful
Some more intuitional explanations, motivations, discussions with pears
playing around with Neuronpedia was good to show us how poorly mech interp techniques can work. I didn't find any of the techniques particularly convincing, and maybe that's a good thing.
Julian is a very good lecturer and that helped to understand new and more complicated topics
The initial discussion was well motivated, and examples given were useful.
The first half of the day established the basis of the field very well .
It was a good intro to Mech Interp, covering a broad range of ideas, older and newer. It’ll definitely help anyone choosing to learn more about any topic in mech interp.
The mixing of theory and practice. The exercises were quite interactive, that helped switching. The slides were quite comprehensive and informative, with references etc. Very good job with the slides I'd say
It was easy to find the source articles on the slides
I liked the transformer circuit thinking exercise
Some small exercises with coding part were great. It's valuable to at least sometimes do them yourself without deferring everything to coding agents. Discussions were great as well.
Doing the exercise about the attention with key, queue value adn hearing the explanation of a colleague.
Interactivity.
The coding exercises in pairs were really useful. The fact that we can spend a big chunk of time asking questions is really good too.
the pair programming exercise on logitlens, understanding circuits through concrete examples and trying to understand how queries and keys are combined from different tokens and how positional encoding of the tokens play a role here.
teaching and slides were great!
The teaching.
I'd previously seen most of the material from auditing the ARENA course, but working through the exercise on K-composition vs Q-composition clarified things for me a lot, so that was good.
The pair coding exercise was the most helpful part of the day. It was very nice.

Would improve32

I would have preferred a slightly more rigorous, in depth coverage of fewer topics. Having said that, I think the lecture was great given how much material was covered
I think the pace was a bit too fast today, I think it's just too much material to fit in one day. I was already familiar with many of the topics covered today, but I imagine for someone without that background it must have been quite difficult to follow.
I think more time for the exercises would be nice.
Great content - just felt slightly rushed for 1 day; I think 2 mechinterp days are probably worth it given how central the field is (vs. some of the more esoteric areas covered).
Overall, I thought that the day was well structured and well done. Pretty small thing, but I think I would've enjoyed spending a bit more time on the fuzzy interp stuff, and perhaps slightly less on the early CNN stuff (though I suppose this was already quite short).
Pictures on slides are small and hard to see, but I pulled the presentation on computer so that helped
Seems like reducing the number of slides from 140 to maybe <100 might be a good idea, and then spend more time explaining the most important results. Some parts went way too fast. Usually, if you spend 30 seconds on a slide, its best to remove it. I find the K and Q-composition circuits are better understood when you make each component appear sequentially instead of all at once. You can also draw it on the board and describe what each component is doing. The Q-composition exercise was not very clear.
The material is very interesting, but perhaps too compressed, I would reduce the material (especially on some of the latter part of the lecture) and leave space for a longer coding session! I find that coding the algorithm yourself helps a lot with understanding.
This was a lot of material for one day. I'd split it in two.
It would be nice if the lecturer repeat the questions because you can't hear them well if you are in a corner of the room
As a broad survey it seemed good to me, although of course I probably don't know about anything you chose to leave out. There were a couple occasions where I struggled to follow this "why is it done this way"/"why does this property matter", so possibly there is room for pedagogical improvement in terms of how to introduce and motivate the core concepts? Or possibly this is irreducible complexity and one must start somewhere. Also, it felt like there was very little time for any exercises today. This may be because we asked too many questions during lecture, so perhaps a worthwhile tradeoff.
I think I would skew the content more towards fundamental concepts such as feature and circuits so that they are explained in more details.
Like yesterday, the time is simply too short. This is worse than yesterday as we had a full day of teaching.
1. The position of the laptop that connects to the monitor ends up forcing the lecturer to stay in front of the slides, covering them. The laptop stand should be moved to the side, to the right most likely (the whiteboard on the right can be moved even more to the right as well). 2. Needed more mixing up theory and practice. The interludes with neuropedia were great and gave an opportunity to switch the brain from theory to practice. 2.5. I felt lost at the long SAE section where we did go in depth of too many approaches. Perhaps a brief overview of each would be enough, as otherwise gives mental overload. 3. J-lens wasn't represented on the slides, only mentioned briefly in the end. 4. The breaks should be separated by 45 mins tops. I think a survey / poll on Discord or WhatsApp would be great about the maximum acceptable interval between breaks. We might also create a button for breaks -- during the lecture, if people feel tired they can "push" the button, and once it gets pushed by more than 33% or 50% of the audience, we have to stop in the next 1-2 or 5 mins. 5. To add on the subject of breaks: right now we have 2.5 hours before lunch and 4 hours after to study, which makes it imbalanced and calls for a big break (at least 20 mins) during the afternoon session. 6. In-depth explanations should employ the whiteboard more, or be more example / exercise driven I guess. As it gets literally hand-wavy, but that's perhaps the hardest to improve
Some places around the class are not well suited to see clearly what happens at the board so if possible it would be great to find a disposition that allows more people to sit in front of the teacher.
I would spend some more time on the circuits in transformers and look at some explicit examples on the board. Exactly how the features were used in the IOI or ACDC, the final attribution graph circuit and the motivation did not feel clear. I would also have benefit from some more explicit (low level) examples in the identifying SAE features section, though I'm not sure how feasible that is. I didn't find the exercises useful in understanding the presented material, more for understanding how to code various bits (which is fine if that was the purpose).
Even though I knew a lot from the material today, I find the pace uneven: too much time is spent on simpler or older concepts, while more complicated or/and newer ideas and methods are rather skimmed through. It is not clear what is the pedagogical goal of the day: is it to give a historical perspective of the field? to explain the basics? explain the latest developments? I think that stating a clear priority would help with shaping the material. Going quickly over complicated concepts towards the end of the day increases mental load and disengages from following. It might be beneficial to simply go slowly and drop the volume of the material. Then, even though the excitement is clearly communicated, there's always this thread about uselessness of mechinterp. I understand that it is reasonable to give a realistic assessment of the state of the art, but it demotivates quite a bit.
Non physicist opinion: be more mathematically precise or tell me when you are not attempting to; define all notation on slides (x_i: what is the i?) Include more practical exercises like the transformer thinking exercise The coding exercise in the morning was bad: the google notebook was too slow, and i mostly struggled with syntax (which felt artificial and not helpful to learn) General: 1. Maybe try to screen participants more strongly for caring about AI safety? I feel that some are just here because of trying to ride the general AI hype wave (not crazy sure about this, just an impression). 2. facilitate more socializing (city tour, etc)
A bit more attention to universality and corner cases will be great. I suppose it's fine to introduce most known concepts / approaches and skip some (like cross-coders etc.), and leave them as links for interested people to explore. What can be most valuable is sharing knowledge about some corner cases, limitations and reasons methods might not work instead. I.e. something that is hard to extract and summarize from papers and which requires lecturer's experience and opinion. I might be slightly biased as I'm at least familiar with the field, and I suppose that perspective of someone less familiar will be different.
- Put the course material in one channel, and then in the messages mark the day and the topic. - Be clear what mathematical topics will be explained in detail and if not where to find resources on the ideas. Some of the formulae and proof I could not follow.
Dense slides, hard to follow if you don't already have a reasonable background.
Better than yesterday. Teacher seemed a little bit more prepared.
A lot of material was covered, sometimes at the expense of depth. It was not always clear when results were robust or cherry picked.
I personally need two weeks to do this two day bloc, and it would still feel intense. Perhaps a series called "Iliad Fundamentals" could also be introduced, where each run lasts two weeks and does an intense deep-dive into specific topics covered in the Intensive.
More paired coding exercises. These were super valuable. We also had a decent amount of time to complete this so it didn't feel rushed.
more hands on things and i liked the parts where students are asked to explain something.
- overall very good, would be nicer to have easier access to slides, notes, etc i.e is the https://iliad-team.github.io/iliad-intensive/ website going to be updated? i think something like this which has direct link to readings, slides, exercises, notes would be nice - teaching was very well done
Overall I think Julian did an amazing job of covering a lot of material in a short length of time and the teaching was great. I think the demos could be improved as it was not super clear how to use e.g. Neuronpedia. Perhaps instead of breaking into pairs this could be demo'd to the group as a whole (and still be interactive for people to guide the prompting etc.).
I think the lecture (and lecturing!) were really very good - although I can imagine the pace would have been very difficult for someone not already familiar with the material. I think some of the precise mathematical statements of results could be corrected. Naturally the statements have to be abbreviated for clarity, but for example, the statement on the Compressed Sensing slide (s58) is clearly false as written.
I enjoyed the content we covered today but I felt like we went too fast. Especially during the second half of the day I felt like I was accumulating lost-ness. I really enjoyed the pair coding and would love to have much more time dedicated to exercises (not just the breaks). I feel like my capacity for new content will grow as we go through the month so I think I would benefit from less content, more examples in the first week.
We went really quickly through the slides.
The redirections to neuronpedia were great. It might have helped to include at least one slide explaining what to look for. Same applies to the many graphics (which looked awesome btw).

Changed my mind17

not too interested to mech interp anymore
I have a clearer idea of what is mechanistic interpretability and I find it really interesting as a potential line of research I would like to work in.
I am new to the filed and I found the topic interesting and worth learning about
Mech interp seems less promising than I thought. SAEs seem pretty useless, most features looked uninterpretable.
I realized probes could be a very promising direction to study.
I have a much more detailed understanding of what's up with mech interp now, but I think my overall feelings about it are roughly the same.
Maybe was a bit discouraged that mech interp will be “successful”.
I did note down quite a few insights, but I am spent for today, no tokens left
I have a clearer view of what we call a feature in MechInterp
It might be that I underestimated some method's promise, and i.e. NLAs, if done properly, might be useful overall.
No
I previously had the impression that mech interp was more rigourous than I do now. A lot of techniques seemed to work in some instances and fail in others, without a clear explanation of why.
I am not super convinced on the utility of mech interp for safety but I think it's valuable that someone is pursuing this.
became slightly more pessimistic about mech interp but more about the other parts of ai safety coming up next :)
n/a
I was a little surprised to hear about how unreliable mechanistic interpretability methods are, which makes me a little less likely to consider research in this field as opposed to another subfield of AI safety.
I discovered the sota of Mech interpretability, its limitations and directions. Maybe worth considering for my future career.

Other thoughts7

Today's lecture was much better paced, and pitched at a better level.
nope
I had a hard time finding the slides. It appears one of the two links to slides on this page was broken: https://iliad-team.github.io/iliad-intensive/interpretability/mechanistic-interpretability/ links to both https://raw.githubusercontent.com/iliad-team/iliad-intensive-C.2/main/lectures/output/main.pdf (404 error) and https://iliad-team.github.io/iliad-intensive/downloads/mechanistic-interpretability/mechanistic-interpretability-slides.pdf (loads, but I think was missing some slides vs the lecture)
This is actually such a great experience, I love being here among the peers. Thank you guys for making this happen, I hope it goes smoothly!
it would be nice to have more desk space, especially orient the desks so its a lecture form and they all roughly face the front?
n/a
Apologies for arriving late today! It was an unexpected delay resulting from my train being redirected, but I'll set off earlier from now on to avoid it happening again.

August 2026 Mon 17 Aug Brianna Grado-White 31 responses energy 3.8/5

Rated against what each person expected. 0 is as expected.

Overall +0.10 as expected

Content +0.19 as expected

Clarity +0.45 better than expected

Motivation +0.55 better than expected

Exercises 0.00 as expected

Readings +0.48 better than expected

Other activities +0.25 better than expected

Pace -0.16 about right

Difficulty -0.26 too easy

Novelty -0.42 mostly review

Read what they wrote44

Worked well13

Feel like compared to other days I understood better how the results relate to mechanical intepretability. The TAs were also good at explaining and also visiting each group and filling the gaps.
Brianna's willingness to answer all the questions and keep refining explanations until everyone was on the same page. Also shout out to Dan (participant) who has mentored ARENA before, and was able to jump in for adhoc transformer refresher.
Readings were chosen well. For people familiar with mechinterp, reading the critiques helped them understand what is missing in mechinterp and what we can hope to get out of it.
It was fun to discuss the readings with other students and the instructors
interesting readings, handwritten slide formatting nice, able to spend some time in lesson and other time not in the room
I ended up reading the posts on "new" ARC agenda that I wasn't aware, that was super cool.
It was very useful to understand how sparse autoencoders can deal with superposition.
I really appreciate the reading list that Brianna curated. I also genuinely want to hear more of the teachers’ takes because that is usually the best part of the day. I think Brianna did a good job presenting the information. I also really enjoyed the comparisons to modern mechanistic interpretability work. Regardless of my own views on that work, it is generally useful to know about it.
Readings.
The content included a mix of readings/presentations for people of different backgrounds and interests: - Some content went over transformers, other content went straight to mechinterp. (I'm aware that transformers were technically a prerequisite for the Intensive, but this wasn't well advertised in my opinion, and anyway there's a difference between reading about it once on short notice and actually grokking it. I benefited from this morning's presentation on them.) - Some content was more empirical, some was more theoretical. (I'm instinctually drawn toward theory, so I appreciated the readings on ARC's theoretical mechinterp work.) This made it easy for me to have an interesting and productive day.
The exercises, the instructor was very good about answering the material and today's lecture was not overloaded. People had good questions and there was more collective answering, perhaps because the avg student seemed to know more of the material and could jump in to help the more confused individuals.
Some of the code exercises were actually manageable and they made me close some percentage of my empirical gaps.
I think the mix of coding excercises and lectures was well balanced. I like that a good amount of time was spent actually trying to make sure we understood the content properly

Would improve17

Brianna's presentation was good, so this is largely meant as a "content choice criticism". I was pretty bored today, as I knew most of this already (and I guess several people did) - only once we did the readings from ~3 PM did I feel that things got fun (reading e.g. ARCs agenda). I think given how different the background level of different people is, perhaps the best way to do a day like this is "Group 1: People who have use of a lecture; easy exercises", "Group 2: People who know the basics, get reading & exercises", "Group 3: People who are quite advanced, get more advanced readings & open-ended questions". I think this is something that I think would be useful more generally speaking on days where one expects people to have varying degrees of knowledge about a topic
I am pretty content with the material. Though, some people seemed to need a refresher on transformers etc which staggered the whole presentation a bit.
Some engineering pre-exercises would be super useful! I was surprised some people had trouble understanding what activations/weights were.
Having us divide into groups and have do group discussions and pair programming would help.
The lectures were bad - skipped over far too much stuff. The readings seemed to not require much technical understanding nor be insightful into mech interp. Maybe it was just that I was tired.
Less focus on the transformer architecture, maybe more focus on CNNs since there the architecture is easier and the feature -> circuit -> universality claim is more clear.
I struggled the most with today's material, mostly because there were a lot of concepts (eg details of the transformer architecture, or mechanism of a sparse autoencoder) which were more or less new to me and I did not manage to put all the info together fast enough to follow the rest of the class
many different experiments and diagrams are on the slides, sometimes difficult to understand without more explanation of the experiments
Since this intensive has less days, maybe not putting this day since I'd guess everyone should already know about it (I mean, if someone is wanting to get into AIS and never read even the basics ideas of interp I'd say that's on them and not on the program. You could say this day is actually useful because people code ARENA notebooks but then this was in general not incentivized by the professor, and I also believe you guys weren't expecting people to learn py torch or ML engineering from this day lol). On the other hand, I think this is mostly my beliefs since a bunch of people from the class seemed to be very confused with basics of transformer architecture, so maybe this day was needed - despite very suboptimal. If the day was natural abstractions and/or ARC current "new" agenda it would be so much better, for example.
- Nit-picky, but maybe reconfirm the essential prerequisites the night before - Replicating some simple toy models might be helpful?
It would be nice for the ILIAD Intensive course overall to start with today’s material, then move to Mysteries of Deep Learning, and then SLT. I think it is also important to begin with an explanation of MLPs and Transformers, regardless of people’s backgrounds, to give everyone the same grounding. I found Dan (the participant who taught ARENA)'s walkthrough of Transformers pretty helpful, and I liked the selection of papers that Brianna curated. Yangdai also gave a really good explanation of the residual stream and Transformers. It might be good to have a manual pencil and paper style walkthrough as exercise as well.
Assume knowledge of DL concepts.
I feel like it was good to see some of the concrete steps, but I feel like a lot of the concerns with AI safety/ examples of unsafe behaviour sort of happens through more advanced training (RL) or something more involved. To me, this gap makes it harder to be 'inspired' for new ideas relating to safety since it feels like today's material, or maybe MechInterp itself, is further from the research horizon (I could be wrong, but it seemed like this to me). I do not think the material should be cut since it is foundational and it seemed to really clarify a lot of the 'blackbox' for many people, including me, but I wish there was some time dedicated to 'looking ahead' or explaining where the field hopes to go from here. Possibly the reading is supposed to help with this, but I started with the reading titled 'Against Almost Every Theory of Impact of Interpretability', which maybe coloured my perception of MechInterp as an approach.
It's challenging to provide honest review here. Today was bad, and it wasn't the instructors fault. It was on us, the participants. It was impossible for the instructors to do anything meaningful today, because we simply don't know what a transformer does. To make things worse, we don't care about the fact that it was a clearly stated prerequisite. Sadly, most of today's readings and exercises couldn't be explored properly as a group.
More clarity on background. For example, I spent a long time trying to learn about transformers but in retrospect, I think I could have pretended everything was an MLP without that much loss.
I think the main failure mode of running this day without the engineering day before it is that a good chunk of the cohort might lack the relevant PyTorch fluency or general empirical experience to follow or engage at all times. Given the compressed format of this run of the intensive I cannot really say I have any ideas as to how to improve it. In my case, even though I had studied the Toy Models paper, I got a bit lost during some parts of the presentation and definitely just because of lack of empirical hands on experience with SAE-like tools.
I feel like maybe a little more emphasis on the problems with SAEs would have been good, so people don’t come away with the impression that they are still like, the standard tool everyone reaches for since they really have fallen out of favour in the field as far as I can tell. However, other than that all the basics were covered really well.

Changed my mind11

I forgot how interesting mechinterp was
Critiques of mechinterp helped me understand the shortcomings of this field and made me realize that it is ok to question the value and methodology of a field even if the field is popular
Nope
Interpretability seems to be mainly an architecture thing; so CNNs are interpretable and transformers are less so.
Mech interpr is criticised, but it seems narrower definitions of it (eg focussing on a single feature instead of mapping them all) is useful
Found out more about open problems in mech interp worth pursuing
Apparently, I conflated deep learning theory with mechanistic interpretability. I didn’t realize that mechanistic interpretability is quite narrow.
Hardness of mechinterp.
I am more convinced that 'steering' and other safety related things (e.g. situational awareness) can be detected by MechInterp
At some point I will have to sit down and train a transformer from scratch, me and Claude against the world.
Not really I’ve done quite a lot of mech interp already so much of it was fairly standard.

Other thoughts3

Another highlight of the day was talking to Yangda about his journey into ILIAD. He connected me with Daniel Chiang, and we have since had many wonderful conversations about agent foundations, as our work overlaps quite a bit. More generally, matching people with researchers working on topics they are interested in early on would probably be a good idea for the next iteration of the Intensive.
Second week off to a great start!
N/A

June 2026 Fri 19 Jun Julian Schulz 10 responses energy 4.2/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.4 of 5

Teaching and format 4.3 of 5

Difficulty match 2.7 of 5

Energy 4.2 of 5

Read what they wrote9

In their own words9

The instructor was very energetic, which was pretty nice. I thought too much was covered too fast, and stuff that I did not know was very hard to understand. I don't really get along with long lecture sessions. Also it was very clear that the content that was being covered was not being done rigorously, so there were a lot of gaps. I can particularly say this because I knew some of the stuff already. This is a clear consequence of trying to fit in too much in a few hours. If that was the motive, alright, because I am clearly going to be covering the entire thing now properly. It was very nice to get the exercises as well, I can go through them later to help understand everything. In general, I hope we have interactive notebooks for every session.
Very interactive, good exercises
Today was my favorite yet!
Great high level overview. It covered a lot of ground giving enough detail to be intelligible but without getting too sidetracked by details. Having a lecture interspersed with occasional exercises worked great. I had a moment to gather my thoughts and make sure I was following. Also switching between these 2 modes made me feel less fatigued. Overall, it was a pretty interesting day!
Today we reviewed the main concepts of mechanistic interpretability used in AI, and discussed some implications in AI safety and alignment. I found the content very interesting and new to me. I'd maybe try to shrink the discussion and focus on fewer conceptual points, because we didn't have enough time to spend on more complicated methods and approaches. I also enjoyed the exercise, but unfortunately we didn't have time to go through all of them even though they were interesting and could have helped me grasp the theory better. To conclude, a more balanced split between presentation and exercise would improve the module by a lot.
Initially it seemed like today was not going to be a very fruitful day till we were doing LogitLens and SAEs because I felt like this is stuff that anyone with interest in Alignment has seen many times over. But after lunch, we did Crosscoders and Transcoders: it was good to get up to speed with the methodological developments of the last 18 months, and get an idea of how the empirical alignment research landscape has progressed, especially NLAs and Activation Oracles. We had the Dan Murfet also today so we didn't get time to do exercises but having time to play around with CLTS, AOs and NLAs would've been helpful.
did not attend the first half (know the basics of mechinterp), second half was well-done and I learnt some things.
The lectures were fabulously communicated and quite well-designed (aside from some timing issues which caused us to skip a few exercises). I enjoyed the circuit tracing content and have appreciated a more formal approach to it. Dan's talk was cool.
I liked the detailed lecture and explanations. Also the topics were in my opinion quite useful and interesting. Also there were exercises which were also helpful.

April 2026 Tue 7 Apr Julian Schulz 13 responses energy 3.3/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.5 of 5

Teaching and format 4.1 of 5

Difficulty match 3.4 of 5

Energy 3.3 of 5

Read what they wrote33

Worked well13

Pair programming, much of the teaching content and demos (generally really enjoyed it!) The regular breaks and exercises are very important because it seems really hard to keep absorbing new ideas after an hour or so
1. 10 min break! 2. Time for independent experiment/exploration/reading then re-group for short discussion.
We got a good overview of MechInterp techniques.
Lecturer so good
I think Julian just did a really good job
The lecture content and resources was really valuable.
Lots of breaks, discussion time, mix of exercises and lectures (time to think about materials/slides in lectures)
I liked the format, which included programming, discussion, and lectures
The presenter was great and their content was engaging. The online tools to play around to find out what's happening inside LLMs was a great way to better understand the mechanisms. The resources provided, even the papers for a conversation at the end were interesting.
The mix of theory and exercises was nice.
The lectures were great.
Julian is engaging and capable and impressive
the interactive questions with tier 1 and tier 2 worked well!

What they took away11

Clearer views of SAE methods and circuits
1. neuronpedia 2. sparse autoencoders breakdown
Circuit Traces (and techniques for finding circuits in general) + the (unfortunately too brief) discussion about arguments "against" MechInterp
Very low level structure of mech interp techniques
Saw the technical details of mechinterp approaches
ACDC
SAE implementations and variations
Playing around with the SAEs and circuit tracers was informative (e.g. to directly see how messy it is)
How LLMs organise related information "close to each other".
How SAE’s are trained and how circuits are detected
I collected a lot of links to read later! I felt I got not only a clear snapshot of the current state of play, but also the recent history of how we got there. I feel I gained something like dialectical fluency, i.e. I now know the meaning of lots of proper nouns I might hear tossed around when chatting with other researchers, whether I like ever use them myself or not.

Would improve8

1. having more time for coding problems! although this would not apply for the days going forward. 2. maybe have a buddy system: pair 2 people with each other to discuss the materials learned up to now (~after 3-4 hours?) => they can resync and reflect on the knowledge better, ask and answer trivial questions, and deepen their understanding to ask smarter questions to the instructor!
More time for the discussion about the usefulness/ToI of MechInterp
It was quite good, I would just have loved to have more time to prep, I could not follow most of it
There wasn’t much time for the discussion or exercises. I think I would prefer to just focus on the lecture content and have breaks in between rather than squeezing in exercises.
more time for discussion ~15-20 min is good for a few topics, shorter is fine if the content to discuss is more limited
The classroom layout sometimes makes it hard to see the board. Otherwise I would like to see some live coding of the exercises (the notebooks), so that we can better connect the theory to practice.
Maybe more time for coding exercises
I found a day-long whistle-stop tour uncomfortable. By the afternoon, I was fogged out and not taking in new information. I felt like I was trying to look polite while waiting for it to be over, so I could read a few of the many papers that flashed by and follow my own train of thought about the ideas in the them. I want to make clear that I have a lot of respect for Julian and valued his efforts! I love that he made a doc for us with the slides and papers and exercises in it, which will make it so easy to dive back in later.

Other thoughts1

I think generally discussion in the sessions should be more limited and mainly just focused on asking clarifications of what is said - I felt probably too much time today was spent on discussions better done more privately

Multi-agent Dynamics 1 run 27 responses, 28 written

for pasting into Claude

October 2026 Fri 23 Oct Aden Power 27 responses energy 3.5/5

Rated against what each person expected. 0 is as expected.

Overall +0.26 better than expected

Content +0.30 better than expected

Clarity 0.00 as expected

Motivation +0.37 better than expected

Exercises -0.09 as expected

Readings +0.20 as expected

Other activities +0.13 as expected

Pace +0.26 too fast

Difficulty +0.15 about right

Novelty +0.67 mostly new

Read what they wrote28

Worked well8

The mix of various speakers and the intuitive approach
I liked the subject matter of James' lecture the most. All 5 talks were impressive.
It was helpful that different lecturers have their own expertise. Topics are various but coherent. Interesting topic and discussions.
I enjoyed the multi-agent RL talk!
i loved that the day was so well curated with a coherent programme±
Final discussion
The exercises with accompanying reading material really helped solidify my understanding today
It was best when they slowed down for questions and encouraged audience engagement.

Would improve14

The remote speaker was a bit hard to understand, both physically and content-wise.
Bad slides, not helpful or informative to read them during the lecture. Better preparation overall would be nicer, the speaker just read off the slides all the time and they were not written well.
Not teaching related, but the room was so hot today. Some fans would help a lot
I thought this was a really interesting topic, and particularly enjoyed James Sykes' talk. I would have really liked to see more through the lens of condensed matter physics. I'm now significantly more interested in this topic than where I started.
More exercises
Very good set of lectures. The lectures were organized well and covered a lot of topics. The lectures were clear and the discussion session a tthe end was especially good..
Today's lectures are very nice, and the exercises are good too. Nothing that I can think of for improvement, just wish today is not the last day of the week where we get a bit tired.
The online talk was a little confusing and very detailed
Make sure beforehand mic and the screen work.
I was a bit under the weather today, hence missed some things, my main gripe would be it wouldve been nice to have more of a concrete connection between the talks/cross references. But it was in general very nice.
two of the talks (James and the second one by Aden) went a bit too deep on the formalism and maths and specific examples and not enough about basic formalism (the Aden's first talk was great! but did not prepare us for James) and general ideas.
Not much, maybe slightly less lecture time. Also fix the AC
The lecture needs to spend more time getting us familiar with the notation before sprinting through successive math equations. It is hard to context-switch day after day between different conventions across different fields and coherently follow things. I think comprehension would be greatly improved with just a tiny slowdown in the beginning of the lecture.
I thought it would have been perhaps been better to start with the real world examples and discussion around them earlier in the day. Having it before the theory stuff would have put the theory in more context I think and perhaps easier to follow.

Changed my mind6

I learned about deal-making - fascinating!
This seems like a more interesting topic than I realized.
This subfield seems particularly new that there are not well established theories based on math or physics at this moment mostly empirical and engineering which could be seen as open opportunities or exploratory. An important field which could benefit the most from contributions in cognitive science or computational psychology aspects imo. Exploring with RL concept was particularly intriguing.
Enjoyed thinking about game theory, might investigate further
I changed my views on studying cooperation and altruism as emergent and defined behaviors in systems. I thought it had to be a product of behavior but I am now more open to mechanistic or training-based answers.
Interested in what the orthogonality thesis means. Realizing I may have misunderstood it.

Mysteries of Deep Learning 4 runs 93 responses, 130 written

for pasting into Claude

October 2026 Tue 13 Oct Charles Renshaw-Whitman 37 responses energy 3.8/5

Rated against what each person expected. 0 is as expected.

Overall -0.66 below expected

Content -0.17 as expected

Clarity -0.40 below expected

Motivation -0.20 as expected

Exercises -0.05 as expected

Readings +0.06 as expected

Other activities -0.15 as expected

Pace -0.14 about right

Difficulty +0.17 about right

Novelty +0.51 mostly new

Read what they wrote44

Worked well11

the reading material seems really good and structured and i am very grateful for it
Serendipitously, I had wonderful discussions with other participants that helped develop my inituitions for how I fit into AI safety.
I greatly appreciated open discussion with Charles.
There was a lot of material, but it was well organized.
Having time to read and work in groups to digest the material was nice.
The readings and resources on the website were very well organized
The Github page is nicely made and provides a good backbone for individual learning (although, as mentioned, I would have preferred a little less individuality).
The assortment of the readings on the various mysteries of DL.
Nothing really honestly, it was just read stuff by your self I could have done it at home
I like being allowed time to read the material thoroughly myself
For the whole day I read one of the suggested papers and reproduced one of the results with the help of conding agents. That worked out.

Would improve25

while the resources made available are extremely valuable, i really do not think that this format of "go and read whatever you want from this way too big corpus" is a good use of our time. i would have preferred some of the material be presented. Mysteries of Deep Learning, Training Dynamics, and Singular Learning theory are the days i was most looking forward too and I was very disappointed to hear that today was going to be this format and VERY disappointed thinking the other days are going to be like this
i think there needed to be more direction for mostly unfamiliar material
In the beginning of class, there was a brief discussion of what each section was about: I would expand on this a bit and give everyone a bit more of a sense of the major papers in each section.
I appreciate the collection of resources provided. Lecture format would have been preferred during the day.
The morning session was a bit unorganized. We basically had to explore several topics ourselves. Would be nice to have organized lectures around the most important topics so that we could at least get the high level ideas.
More structure given the fact that the topic I very broad. I don’t mind reading but I prefer it if it is aimed at something specific.
Though I appreciate the thought in a self-guided philosophy, I would strongly prefer a greater structure and greater input from the instructors. I can read curated papers and ask LLMs for advice on further curation in my own time. The only benefit of attending today was the presence of other participants. Note that I had trouble answering questions about pace, clarity, and difficulty given the absence of pacing, explanation, or challenges.
The format of the day could be improved, in my opinion. Specifically, I would have appreciated a more selective list of articles, with some structured discussion groups among the other participants (and Charles and Brianna). Ultimately, I find it much more useful to engage in discussion than quietly read articles, to the extent that the discussion doesn't dwell in too much bullshit.
It was all self reading today, so a heads up on Discord would have helped prepare us better for that.
I am a self-directed learning aficionado and I appreciate the github repo, but this is not what I traveled to SF, and commuted to Mox, for. I paid someone 1k today to do my job at 5 just to have the flexibility to stay here, and I am not sure for what. At least you could let us know in advance that's the format. The insane amount of material was also discouraging. It's the responsibility of a teacher help students make the most of the time when the amount of material is too high. It is one of the very unique few things an instructor can provide.
A bit more structure to the day. It would have been nice to have 15-min check-ins with a brief intro or overview of the modules during the day. The readings were also a lot; the check-ins would have helped highlight what was important to get and their relevance to later sections.
I would appreciate a high level overview of each of the 3-4 mysteries before diving into the readings. I think it remains important to distill some intuition/opinions to overall get a sense of direction or engagement with the research questions before diving into it ourselves.
I am not a big fan of the 'just read some stuff' approach. I personally need a more concrete approach and a stimulus to engage in the material with fellow students. The current approach made me drown in the freedom somewhat, which is a shame, because the material is genuinely interesting.
Might try a little more structure to the day. Perhaps two 20 minute talks to anchor the rest of the day's readings.
I understand the motivation behind giving us a repo for self-study, but what I value most in a lecture is a clear throughline connecting the most important ideas, even if that means being selective and leaving some material out. I was looking forward to these lectures as a chance to hear how researchers in the field think about the topic and which ideas they consider most useful. Discussion and questions are valuable too, but I think a prepared lecture gives the speaker more room to develop their perspective in depth than responding to questions on the spot.
Unfortunately, the lecture did not really take place. I believe many students, myself included, were genuinely looking forward to it
I think a general structure was missing and a better outcome could be achieved with more structure.
Make it a class
not much maybe a few lectures but i like self reading
Maybe a Bit more teaching in the classical Sense, content was good
Give an overview of the ideas before delegating the reading
Unstructured reading time does not feel productive, that is something we could have done remotely (outside of the intensive).
Required more structure: free-form from the get-go left a lot of people confused. Perhaps spend the time before lunch going over some basics and giving us some intuitions before setting us free.
It would be great if we get a general overview of the topic during the initial hours: The materials were good and the session gave an opportunity to dive deep into the topic of interest. Charles and Brianna were around all day to answer all questions. But a s a person who was new to this field, reading 'alone' was not really helpful to actually uderstand the topic well. I would recommend strongly to include bit more lecturing hours - say 1 hour in the morning and maybe 30 mins in the afternoon. This would help us steer into the reading sessions.
The formation of the lecture. I would have liked much better if the lecturer provides substantial materials explained.

Changed my mind6

I think I am convinced that research into aspects of deep learning or interpretability for transformers are not for me. These directions do not address fundamental concerns I have (total loss of control, gradual disempowerment, bad actors), while being less aesthetically pleasing than other research opportunities.
I'm doubtful Singular Learning Theory is a promising approach to native LLM ontology.
I was better able to understand why even measuring generalization is a difficult problem and how counterintuitive deep learning can be on that front.
I became a bit more aware of the magnitude of stuff for me to read/learn and it increased my drive to dive into things.
Virtually everything was new, so I did not change my mind.
No

Other thoughts2

I think the self directed learning approach was ineffective. There were too many readings to chose from and I didn’t know what to prioritize.
Please try to implement the suggestions about lecturing hours pleeeeease!!!!!

August 2026 Fri 14 Aug Zach Furman 32 responses energy 3.9/5

Rated against what each person expected. 0 is as expected.

Overall +0.41 better than expected

Content +0.50 better than expected

Clarity +0.59 better than expected

Motivation +0.81 better than expected

Exercises +0.33 better than expected

Readings +0.38 better than expected

Other activities +0.50 better than expected

Pace -0.19 about right

Difficulty -0.22 about right

Novelty +0.19 an even mix

Read what they wrote42

Worked well9

I appreciate when Zach just tells us his genuine opinions because it is indeed the most useful parts beyond whatever reading materials. I wish that the whole day is just pure genuine opinions even though it's niche because it would make me be able to assess and think for myself better.
the whole premise of trying to understand why scientific understanding of deep learning is ACTUALLY hard and how empirical developments led to a complete paradigm shift in learning theory. working through the relevant concepts like (un)learnability and expressivity and the gap between them, what kind of loss landscape geometry you have around hard to learn functions etc was particularly instructive
Forcing group discussions worked!
Lectures
some interesting topics, interesting additional readings
Loved the discussion
I think the discussion format was well-suited for the task at hand.
The program synthesis article has a very interesting hypothesis that I wasn't aware of, and the approximation article was a very helpful read.
The readings followed by discussion worked quite well together

Would improve20

I didnt like that what youre left with in the class is like "yeah deep learning is weird, dont be fooled to believe statistical learning theory works or whatever", but the experiments that show that cam be explained quickly, i feel most of the class shoulve focused instead on explaining the approaches to explain these problems (and how they fail)
I think Eigen hall is a little too cramped for all of us. It would be best to use Cantor hall generally. I think Zach is a good writer and a good curator of relevant resources.
Don't overcomplicate and spend too long on a similar theory
today worked really well actually i can't
A lot of "qualitative" content and discussion that was kind of just opinion-based, rather than something technical. Many of the readings felt like rehashes of the first one, or of things said on previous days. Moreover, the discussion felt somewhat but not very novel, and it seemed like we often moved on before discussing things to the satisfaction of those who raised them. I think it would help to use discussion formats with some kind of break-out allowing small groups to discuss and debate in detail (and this needs to happen _after_ a round of reading and sharing takes or questions with the whole group, as this generates potential discussions).
this was a discussion about the broad landscape of research. I wish it had focussed more on safety prospects and not only on the science aspect
Scheduling it first would be better.
I think today was too vague; I think it is hard to internalize or learn things without doing something concrete. I get that this was more of a high level meta overview day, and maybe it is just hard to feel the learning that is happening, but I fear that I have learned to say something that sounds correct without properly grokking it.
Perhaps more freedom on what to read? As in "pick yourself what to want to read, XXX will be discussed from "10:10 -> 10:20", YYY from "10:20 -> 10:30" etc.
This would make more sense earlier in the schedule, but I assume you already know this
It felt like a lot of repetition. Maybe it would be good to make this 2/3 or 1/2 of a day, and do less discussions/cut even more of the readings we actually do today. The program synthesis blogpost felt like a repetition of slides+things in other papers.
crash course on statistical learning probably; the way that it seemed like computational mechanics was a crash course prior to world models and transducers
Have more mathematical exercises in groups
Some of the discussion questions were too easy and uninteresting to talk about.
imo the best format is to have us disperse from the lecture hall, form groups, and do a few readings and discussions at once. having us all in eigen continuously is cramped. it's difficult to form groups, and many people closeby were not interested in discussing
I was quite familiar with the content. I would love to have another chance to dive deeper into some theory like SLT
Better than I expected, mostly because I expect these conceptual days to be kind of boring, but the blog posts were actually great, didn't read most of them previously, nice. Given this context, the part we read 5 pages of a book without context/access to some definitions in the book was kind of hand wavyy, so maybe write a small blogpost with that content contextualized and use it - but this is somewhat minor. The PRH and induction heads paper are kind of bad tho (PRH paper is methodological crap and induction heads is in mech interp), so maybe some other readings would be better. Also, looking at the April docs curriculum, it is perceptible that the ordering of conceptual days is made in a way that systematizes conceptual stuff, which seems very cool. This day for example systematizes well this "approximation/generalization/optimization" intuition/frame, that was cool, but then the ordering got changed and days were deleted so this conceptualization/framing felt a little bit out of place (as in, it seems obvious that you made the curriculum expect it to connect and be in some ordering and it is not and this is kind of bad)
Today was great, I really enjoyed it, helped me learn a lot and put a lot of different parts together!
I don't know actually for today the reading oriented session felt suited for the topic, and that was executed pretty well so I can't think of anything.
Wish this was positioned earlier in the week

Changed my mind9

I knew that there isn't much understanding of deep learning theory but I didn't know that it was genuinely so little understanding. I also appreciate that Zach and Paolo have some information about the environmental impact of LLM usage.
i now am even more theory pilled than before
The whole "Bayesian learning generally converges rapidly" was something I haven't thought about before
Platonic representation hypothesis seems to have a strong implication for safety if true
I am now at 60% leaning towards grad school (40% AIS). I started at 80%.
I'm less skeptical of statistical learning theory for DL, as long as we throw away VC Dim stuff and incorporate data/sgd path in our complexity measures
in some sense I understood how mathematics could be relevant for a more overarching view of things (rather than for computational aspects), but also seeing a lack of formalization in the objects involved (e.g. 'space of programs') made me feel like the field is really behind, or doubtful how much progress can be made through mathematics if the starting point feels so far behind the theory that could eventualy help.
Why it is difficult to explain why NNs perform very well on approximation & generalisation; how optimisation differs from approximation & generalisation.
I feel like I now I have the main surprises of the modern deep learning era in my pocket to explain to people whereas before my understanding was far more nebulous

Other thoughts4

Continue group discussions
It was a bit hard to read with so much background noise in Eigenhall. Might be a good idea to use the model of Monday wrt discussion (and bring people in every once in a while).
I was confused about some practical aspects yesterday, not because of the instructor but because the material revealed gaps in my knowledge (which I mentioned in my feedback yesterday), but actually most of these were confusions were cleared up after discussing with my peers over dinner.
N/A

June 2026 Tue 9 Jun Garrett Baker 13 responses energy 3.7/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.0 of 5

Teaching and format 3.2 of 5

Difficulty match 3.5 of 5

Energy 3.7 of 5

Read what they wrote15

In their own words12

I thought the readings were good but I would have liked to have time for more mathematical detail. I think we could have spent more time on reading and less on discussion today. The reason I'd say this about today compared to yesterday is that today was a bit more technical so discussion questions were comprehension questions that could perhaps be answered more quickly. I think we didn't need to spend as much time on the intro lecture in the morning.
I really like the discussion format, I feel I learn a lot interacting with others. The questions time after the talk was too long, I would have much more questions after reading the texts. I do not like the military vibe of "you read this now fast". The amount of text we are supposed to read _and understand_ is unreasonable for someone not already familiar with most of the concepts. Perhaps a feature, not a bug, but worth sharing I guess. The questions for discussion sessions are rarely useful or well phrased, sometimes they have litttle to do with the content and ofter they are quiter shallow - this could definitely be improved.
Discussion sessions are not so helpful I think because I feel like we are discussing on topics none of us grasped perfectly and there is no one guiding, fixing or explaining things clearly.
I felt there was some confusion about the initial lecture. There was _a lot_ of time reserved for questions, but this was unexpected for me. I.e. not announced etc
- I was quite happy with the topics covered generally - The presentation in the morning didn't have enough content, or give enough prompts for interesting questions to last the time it did - I feel like the discussions with my group are difficult because we have different mental models of things, but it has the advantage of sharing ideas and learning from each other.
I would've appreciated if the lecture was given in person, something similar to frontal lectures with a black/white board. Amongst the different topics covered today, depth separation was probably the most interesting although I'd liked more details on this.
Good format: lecture > readings > discussion groups. Only quip B.2 came before B.1, and so some of the necessary background was assumed. But all in all, very nice day. Learnt some interesting material today.
The program synthesis framing tied the day together. The most useful part was seeing approximation, generalization, and optimization through that lens, especially why optimization is a separate mystery. The main improvement would be a short primer on the algebraic geometry vocabulary before discussion, since a lot of people got stuck on definitions.
Keeping the same groups was a good idea, as we could refer to earlier points of the discussion. Also the bit of feedback provided by Garrett walking by and eavesdropping was helpful without being overbearing. We all have different backgrounds, but I think it would be helpful to have some baseline technical definitions to make sure we are talking about roughly similar things. But I'm biased towards front-loading definitions and putting together the big picture later, which is the opposite of the pedagogical approach you have picked :)
Discussion groups worked well. The program synthesis hypothesis was pretty interesting. I would have liked the opportunity to run some illustrative demos or a problem to work on maybe.
The talk and discussion period in the morning felt a bit stretched out and could have been compressed slightly. That time was missing later for reading and discussing the papers.
During discussions in smaller groups I'm not always 100% sure if we arrive at proper conclusions for the proposed questions. Maybe we should combine these small group discussion with a "general one" involving all the participants?

Other thoughts3

I am new to the field so it's quite intense for me but I think I'm doing fine so far, learnig a lot.
Not yet
Really valued the small-group discussion format.

April 2026 Fri 10 Apr Zach Furman 11 responses energy 3.5/5

Called “Deep Learning Theory 2” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.9 of 5

Teaching and format 3.5 of 5

Difficulty match 2.9 of 5

Energy 3.5 of 5

Read what they wrote29

Worked well10

I found Zach's blogpost (and the content in general) very good
I liked the introduction and some of the readings
presentation
The food was good, and the office space was nice and well-suited for the starting lecture.
Started out feeling slightly chaotic, but all the concepts fit together by the end, through the readings and group discussions.
Hard cutoffs of readings
I found the content very interesting and useful
First reading from Zach was excellent, and would have been good to have read yesterday (in general, more time with readings from today would have been preferable to some of the readings/exercises from yesterday).
The readings were fun and the tables with 3-4 people works better for discussion.
1. The reading was nicely curated and built our intuition much better. The order for the materials of yesterday and today should have been reversed. 2. Having TA in each group really helped. Overall much more insightful than yesterday for sure!

What they took away11

The deep-learning-as-program-synthesis hypothesis and representation similarity/convergence
That there are different hypotheses for what pressure creates generalisations
range of approaches and results to study the ways models represent concepts.
Getting a sense of why we might suspect that DNNs implement programs, and what that might mean for how to study them.
Perspective of softly enforcing, thinking about program synthesis as a perspective
Group discussions were especially valuable. We debated (eg) whether regularization can account for successful generalization, and I learned about several new papers on this.
Clarified a lot of emerging thoughts I was having
Zach’s blogpost was the most useful reading and perhaps the Platonic representation paper though I disagree with many aspects of it and think it’s counterproductive, good to have gone through it.
First reading was great, I also thought the Platonic representations was also good, just because it referenced a lot of results (e.g. stitching together model representations) that were useful pieces of knowledge to have
Learning that we don’t yet know exactly how models generalize and that it’s debatable on if they are actually learning or not.
Wow so many, the idea of program synthesis, and different representations of that idea: soft inductive bias. How this explain curse of dimensionality, but we don't have low-level mech interp proof -- we need to utilize indirect evidence. Zach's post presented 3 paradox, while the soft inductive bias paper presented other technical underground for this.

Would improve8

More breaks (in the afternoon), more structured post-reading discussions, and maybe more targeted reading (i.e. telling us which sections to focus on in the papers)
Just the long reading does not feel very effective
time efficiency and purpose of activities: it is not always obvious what exactly the take-away goals are supposed to be, and the last two days it has not been obvious whether or not those were achieved. Using Claude makes the summary process easier, but knowing what the instructor wants us to know would be helpful. Sometimes exercises (math or code) are good and sometimes discussion is useful, however individual groups can have an imbalance of discussion content and focus that misses the goals unless a TA provides direction. We spent much of the day repeating concepts from yesterday and some of the discussion didn't really go anywhere.
A bit more variety would be nice, as opposed to just reading. But I appreciate that that is difficult when we just have to get familiar with a bunch of ideas.
Probably the main blog post is a bit too long for this format, so probably a more condensed version would be ideal for this
I think getting people to talk in small groups works better than the big group discussion but it would be ideal to keep mixing up the groups. The big group discussions could also be very useful too but ideally there would be some instructor involved to ask people who don’t speak out much what they think. Currently group discussions are too dominated by the same three people (including me btw, I’ll try to be more mindful to give others space to speak moving forward) which seems detrimental to those who are by disposition more quiet and it would be nice to have the instructor now and then specifically ask those quotes people to share their thoughts. Finally, sometimes when the group discussions do get too dominated by the same people (again, I am partially to blame here and will try to do better) to the point where it is eating into time for the readings or other discussions, it would be good to have the instructor move things forward more reliably. This is not just directed at Zach, I think in general the instructors are very kind and patient with conversations going off-topic but I wouldn’t think it’s a bad thing to not hesitate moving things along earlier.
More time to read, skimming the papers made me feel I only learn the surface ideas which made conversations harder. Also more time for conversation.
2. Small group discussion is ok but seemed too repetitive if we do it for every reading. Maybe after 2 readings => 1 group discussion? 3. can we have more rest pls? 4. can we have Claude Max account pls? 5. some creative ideas: after ~2-3 readings and it's group discussion, have everyone share 1-minute their understanding of the materials. Can employ some techniques like debate, rapid question, or some working together to prove a connected theorem to aid our reflection. Also, the instructor should prepare some guiding prompts (like what Leon did on Day 1).

Physics for Deep Learning 3 runs 84 responses, 102 written

for pasting into Claude

October 2026 Wed 14 Oct Tudor Dimofte 26 responses energy 4.0/5

Called “Physics of Deep Learning” that cohort.

Rated against what each person expected. 0 is as expected.

Overall +1.13 better than expected

Content +0.88 better than expected

Clarity +0.63 better than expected

Motivation +0.42 better than expected

Exercises +0.54 better than expected

Readings +0.67 better than expected

Other activities +0.47 better than expected

Pace +0.25 too fast

Difficulty +0.33 too hard

Novelty +0.58 mostly new

Read what they wrote28

Worked well10

Pacing ws excellent
I enjoyed the lecture format.
The lecture was super super clear, well explained, and nicely paced.
the structure was really good
This content was exactly what I came to the intensive to learn. I wish we had more lectures on exactly these topics. Readings and exercises were well selected.
HELL YES!! clear, structured, well paced lecture on fascinating and difficult material. this is all i wanted from this day and i wished more of the days were like this. perfect mix of lectures and exercises and discussion too
The schedule and lecture materials were well planned.
The excercises.
quality and clarity of presentation
The format of the delivery was excellent: excellent choices were made about what to cover. Also, the exercises generated productive discussion and advanced my understanding of what is going on.

Would improve12

Was excellent
This was interesting in the sandbox/toy setups used to study mathematical properties of NNs/LLMs but I didn't end up with much optimism about its application/potential to better understanding actual applications to LLM research.
nothing.
There are sessions that feel too open-ended, leaving too many things to explore. Would be nice to switch back to structured lectures.
This was the best lecture of the intensive so far. I wish we had this Tudor teaching for another few days. There is literally nothing I could suggest to improve it.
nothing
It felt like the first two lectures were very dense but the third lecture was actually pretty light. I think the time could be distributed so that there was a little more time spent on the same material in the first two lectures and less on the third.
It would be helpful if lectures included explanations of relevance to AI safety & interpretability, both short-term and in the long-term.
The lecture, exercises, and notes were very well organized. Can't think of anything else to add within the time limit for the topic.
It was wonderful lecture!
nothing comes to mind
More time to digest the many ideas present here.

Changed my mind6

I had no prior knowledge of physics inspired neural nets of deep learning. This lecture inspired me to learn deeper into it.
I enjoyed the content today.
I have more hope that a theory of learning mechanics is possible.
Haven't changed my mind on anything really.
I'm slightly more convinced about learning mechanics being achievable at some point in the future.
unsure of how i can fit in when ive only done my undergrad

September 2026 Wed 9 Sep Tudor Dimofte 42 responses energy 3.8/5

Rated against what each person expected. 0 is as expected.

Overall +0.62 better than expected

Content +0.72 better than expected

Clarity -0.13 as expected

Motivation +0.18 as expected

Exercises +0.15 as expected

Readings +0.47 better than expected

Other activities 0.00 as expected

Pace +0.41 too fast

Difficulty +0.38 too hard

Novelty +0.72 mostly new

Read what they wrote62

Worked well20

Tudor was a very clear and engaging lecturer
I found Tudor to be a great teacher!
doing the lab of lazy vs rich models was very fun! Doing the scaling exercises was nice too.
Exercises, lecture before lunch
The presentation -- LaTeX-rendered is a love letter to Physics Academia. The breaks composition -- very balanced, and having a big break in the middle of the afternoon session -- thank you for that. The fact that lectures weren't longer than 1 hour. The exercises. It was a great day overall, I liked it.
I am very grateful that the lecturer answered questions during the breaks. These explanations really really helped my understanding. I also loved discussing the material with other participants. The reading at the end of the day was great. And the discussion was also thought-provoking. I liked the 3rd exercise sheet most.
Clear synchronisation between lectures and exercises. Solutions provided as well.
We had lenghty periods to reread and ponder the materials
I think that the discussions / exercises in between lectures was very helpful (it was nice to have a bit of a breather between lectures)
The reading and the overall picture of the field.
The content is superb. Super interesting! The exercises are good but sometimes I got a bit lost in the bigger pictures, so even if I could do the maths in the exercises it wasn't useful to do them because I didn't get the utility/what it was getting at.
I liked the teaching style. The topic was very interesting.
The exercises were well-chosen.
Pace and amount of material were adequate and not overloading
I thought Tudor was a great lecturer.
More structured breaks, and lecture lengths. Also the pdf of lecture notes and exercises with solutions is super helpful.
- exercises, extra reading, notes (super well put together)
The long break, the different exercise options. The very clear structure of lecture, exercise, break.
I though the lazy vs rich learning dichotomy was very informative.
it's nice there was more actual in-depth content today, but i was lost super quickly (no physics background)

Would improve26

Perhaps take longer time with some of the more difficult concepts like NTK, I found it difficult to follow (without a physics background)
Occasionally, variables weren't explicitly defined on the slides, so stating what they all are more explicitly would be good, especially when notation changes between topics/slides.
I would divide all this day's wroth of content into 2 or 3 days (or even a whole week!), to really go deep and give us actual tools to use from it, as opposed to just having a vague recollection of what happened.
Nothing was a really interesting day
slides are loaded with information that is sometimes glossed over quickly and could gain from more plots to visualize actual research results.
Make the connection between deep learning and physics clearer before moving onto the physics introduction. The physics introduction was a little derailed by people asking questions attempting to connect the (familiar to most) physics to deep learning.
Post lunch the slides were hard to follow
I think notation in the lectures today could be improved. I was getting lost sometimes. Same goes for exercises. The NTK was quite confusing for me and a few people around. That part should be more visual and geometric (just like LR is explained via N-dimensional vector Y being projected onto d-dimensional space of N-dimensional vectors X (d=number of features)). Also some people were in quite a disbelief that this is useful and asking "So what?" and "It's a toy model, how is it generalizable?". A better detour into Physics for how fundamental knowledge was opening the doors to new scientific directions and technological explosions (Quantum Physics, perhaps GTR), as well as giving us profound understanding of things we could hardly define in the past ("features" is one such term).
The material was quite difficult for me. I think it would be helpful if during the lecture we more often drew these connections between the physics theories and neural networks. For example we talk about Ising model, we have these nodes etc, and nodes correspond to X in neural networks. And Y in physics corresponds to Z in neural networks... etc. So to summarize, I was quite lost during the lectures today. But then during the breaks I asked questions and this helped my understanding a lot!
Today was quite nice!
Add more pictures to the presentation, maybe give more intuitions and fewer equations
Content was good, timing was a little suboptimal (limited time for exercises/readings/questions/discussions), I think it would've been nice to have a few more pictures in the slides. For example, there were a lot of questions when the Bayesian learning model was introduced. Here's a picture that I think would've helped me grasp the motivation: A 2D schematic of the parameter space, loss landscape sketched topographically. Upper panel: Widely-scattered cloud of points representing initialization distribution. Lower left panel "SGD": Cloud of points obtained by running SGD for a short time, more concentrated towards minima. Lower right panel, "Bayesian learning model": Cloud of points obtained by sampling e^{-\beta L}, also more concentrated towards minima, hopefully looks similar.
It would have helped to have a bit more details on the slides especially regarding the definition of the notations used
Another great day of content - only limitation was I felt the methods covered could have been slightly better positioned against the wider literature on end-of-learning and learning dynamics.
Potentially, having some option to make notes directly on the slides or project the presentation onto the board could be great, so that the lecturer could derive and explain some things more clearly without hesitation. This is more of a meta comment though.
I would have liked a more basic and foundational introduction to the field, and maybe some mentions to physics for ML apart from statistical physics.
Quite a bunch of regimes and models were introduced and while they were self-contained and clear in themselves, at the end of the day, I still feel a bit fuzzy as to when and how each is applied. Maybe taking like 5 minutes at the end to recap where each fit and what they do well would be useful.
Extremely interesting content which was relatively well delivered. I think the lectures were an adaptation of the content in the paper he shared ("There Will Be a Scientific Theory of Deep Learning"), and he added the necessary context to understand this well. Comments: - The motivation wasn't extremely clear. The motivation is the paper is much better and I think the lectures would benefit a lot from "copying" a bit what they did. - The prerequisities for this day are not really present, eg. I went on a rabbit hole trying to understand the differential geom part of the lecture (which I loved tbf). This can be completely skipped by doing everything in R^n, OR include in the prerequisites the tools used so that we can revise them a bit before. - Overall, there was a lot of content. This is very interesting so I think it would be valuable to have 2 days instead. Also, the slides are too dense. We have a copy of them so extend them and write things fully!!!
Very interesting day. I wish I had a stronger background in statistical physics to understand the classes better.
The teaching and excercises were very good, I have nothing to suggest improving upon!
I think the section on NTKs, while it felt well-motivated and it felt like the material covered was well-designed, was much denser than the rest of the lecture(s) - I'd probably allot more time to this section than we had. I'd say we probably didn't need as long breaks as we had. Also, I didn't really feel the chosen material covered much about phase transitions, although I understood how the Ising model was relevant in other ways.
I enjoyed there being structured lecture notes and exercises I can return to later. I enjoyed today.
very little suggestions apart from improving iliad-team website with more centralised stuff for material. exercises are good, would want more of that. would want more detailed prereqs for the day though, leadup can be hard for non physicists. the notes and readings are great it was just a little fast.
A bit more bigger picture.
I thought the slides could have had more plots on them. A lot of heavy math but not enough qualitative interpretation of it.
maybe: include some option B material for TCS people who are lost on like slide 2

Changed my mind13

All of this was genuinely new to me and I found it super interesting. Will certainly explore and think about these ideas further.
Im still thinking about this
there are more links between physics and machine learning than I expected, but it stays very complicated.
That it is actually more useful and promising that I thought -- it gives us mental models and predictive power for some things. However, it is also much harder that I thought. Also, the paper "There Will Be a Scientific Theory of Deep Learning" underplayed scaling laws -- we do have models for predicting exponents, but how successful they are is a different question.
I understand not that there may be such a thing as a "universal theory of learning mechanics".
Not really; I was hoping for more discussion of how physics of deep learning work relates to safety goals, I don't think my understanding of that picture really changed.
In the end we skimmed through a paper arguing that there will be at some point a solid theory of learning mechanics which I guess would help to consolidate the different efforts thrown into AI alignment so if this is true I think it is quite good (except if it allows even more rapid development of the AI capabilities comparatively).
Maybe because of a bias or a misconception coming from myself, I thought that predicting features of a model at a large scale had to be too unreasonable. However, to my surprise, a physical-like approach was able to be useful to predict some behaviours/laws. I found this particularly inspiring.
I learned a lot about physics of ML. It helped me understand not only physics of ML, but also the inherent concepts of points of view that physics itself is based on.
Yes- I want to have a look at this in more detail.
I didn't really see how the topics we covered today directly impacted AI safety, although I can see how research on these topics could be valuable to anyone working in AI.
A bit of increased skepticism on if the physics toy models are the solution to the AI Safety problem and unsure about how much part of the AI SAfety pie they should have.
It might be possible to theory-model deep learning.

Other thoughts3

n/a
Great teacher. Leveled with us students a lot, was very curious and into what he was teaching! Liked that he went into details a lot.
N/A

August 2026 Fri 28 Aug Charles Renshaw-Whitman 16 responses energy 3.3/5

Called “Physics of Deep Learning” that cohort.

Rated against what each person expected. 0 is as expected.

Overall +0.25 better than expected

Content +0.50 better than expected

Clarity 0.00 as expected

Motivation -0.31 below expected

Exercises -0.06 as expected

Pace -0.06 about right

Difficulty +0.06 about right

Novelty +0.94 mostly new

Read what they wrote12

Worked well2

The freeform study. High level intuitions. And how to think about hard things in an easier way. I particularly appreciate the first lecture on how to think about kernel trick.
I found the part the on NNs that learn features rather than using fixed ones the most useful.

Would improve6

I'm really biased but I really like Charles'style of teaching. The notebooks can be improved and have more exercises.
No more slop exercises. They are demotivating. Quicker start to the day, I think we spent too long talking about the plan and outlining.
I feel like Charles largely "vibed" this and didn't have a clear pedagogical plan here. I felt like this could have been a lot more structured. The note books are pretty solid, but I think notation could be simplified a lot
I didn't understand why I should care about feature function, so maybe that should be emphasised more. Generally I like to learn something because I need it to solve a problem; today I feel like the vibe was that there is a mountain of mathematical techniques and ideas I need to learn to get to the forefront of the physics of deep learning, and I should trust you guys to lay out the path for me. And I do trust you guys, but there might be a more directly motivating way to introduce feature vectors. I think Louis's presentation on data attribution was especially good at motivating why it is important.
Charles is a great teacher and the notebooks are great (though being honest, I fully did the first one and skimmed through Notebook 2 and 3). I think if those notebooks become (more) self contained lecture notes, this would be one of the best days.
I would make the lectures more mathematically rigorous and prepare slides walking the participants through the material. The lectures we were given were pretty unintelligible to me and whenever a mathematical question was asked it was just brushed off.

Changed my mind3

I had a conversation with Daniel Chiang and we disagree on studying rational vs irrational agents and his argument convinced me why we should care more about studying rational agents in mechanism design.
I now think that this is extremely dual use and I don't really see how this connects to safety.
Didn't know why people say NTK is useless, I get the general vibes now

Other thoughts1

It's unfortunate that this is the last class. I'm gonna miss the classes and I'll rework the exercises on the days I got sick.

Policy Gradients and Misgeneralization 1 run 11 responses, 8 written

for pasting into Claude

June 2026 Tue 16 Jun David Quarel 11 responses energy 3.7/5

Called “Policy Gradients and Goal Misgeneralisation” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.3 of 5

Teaching and format 4.5 of 5

Difficulty match 3.0 of 5

Energy 3.7 of 5

Read what they wrote8

In their own words7

It was a little hard to understand the architecture of the code of the morning session. The afternoon session was quite straightforward after the lecture, but maybe it would be more instructive to give away a little less. I'm not familiar with pytorch, and didn't have much time to go over the prereqs, so it was a bit of a steep learning curve, but I feel that I still got something out of the day
I liked having plenty of time to explore with coding. And seeing the theory instantiated concretely. Overall, I'm sometimes feeling it would be good to somehow recap the previous days in some way, but not sure how to do that effectively.
coding in the afternoon was fun. stuff we did in the morning was pretty hard, since i didnt have experience with pytorch.
the policy gradient derivation was clear, especially why the environment terms vanish under the gradient. the best part was linking vpg as the optimizer to goal misgeneralisation as what gets optimized. exercises were a bit easy for those with rl background, so an optional harder track would help.
- I really loved being able to see the agent behaving in a gridworld - Coding in pairs was great for our mutual understanding - It would have been interesting to dedicate some time to discussing RLHF, I don't know if we will do it some other time
Great session, the exercises were very useful for clicking with the theory we reviewed. David was very engaging and keen to guide you through the exercise if you got stuck.
David was a great lecturer; very energetic and gave off a good vibe. Would be great to have him again. The material today was more standard RL, so at times felt like _not_ the best use of lecturer/participants' time as we could use this precious time on a more AI Safety relevant topic and keep the standard RL work as a prereq (as policy gradients are a standard topic covered in all RL courses in Uni). Would be fun to have more frontier RL \cap AI Safety type work (or at least get an intro to frontier work). Pair programming was a good format!

Other thoughts1

really enjoying the mix of theory days and coding days. the pacing between the two works well.

Principles of Learning 1 run 12 responses, 33 written

for pasting into Claude

April 2026 Wed 8 Apr Zach Furman 12 responses energy 3.1/5

Called “Deep Learning Theory 1” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 3.6 of 5

Teaching and format 3.0 of 5

Difficulty match 3.3 of 5

Energy 3.1 of 5

Read what they wrote33

Worked well10

Zach was very calm and clear and humble. His pace was slower than Julian’s, which I liked.
Just a nice overview, more depth would have been ok for me/more targeted exercises or where to find them. And a bit more background on why this is important/historical background of the subjects would be good!
It was fun to have some math exercises to do, as opposed to just reading and listening.
Readings
Exercises were useful and main presentation was good, zach is entertaining to listen to and answered questions well
The lecture at the beginning setting everything up and the discussions at the end were useful for bug picture context, though I think some of the detailed Solomon off readings were a bit too dense to take away much from (the exercises were I think, in the other hand, a bit too easy to be super informative as well).
Some of the more mathematical readings were a good review of central learning theory concepts.
Good presentation and interesting readings, also nice to get hands-on with some exercises
the instructor is very kind and helpful!
The content is interesting and seems to be generally good background material. Doing the maths exercises was nice. The talk this morning was quite nice, but probably more detail on what is deliberately imprecise would be good (perhaps a note to say that a slide is not going to fully make sense, but paves the way for later).

What they took away10

Probably the nicely phrased final reading, 8.4 of the Cambridge textbook, on learning and cryptography as opposite processes
How these concepts tie into AI safety
The idea behind Solomonoff induction, and its relation with simplicity bias. Also the way in which NFL theorems can razor-away certain justifications for the performance of learning systems.
Solmonoff theorem proof
solomonoff induction schematics
It was my first interaction with Solomonoff induction - it was useful to understand what was going on, and especially to get results on universal behavior of various priors.
Solomonoff Induction and prior
approximation, generalization, optimization. the instructor's explanation of Solomonoff induction and its architecture.
Solomonoff induction theorem and some further intuition for what the various background assumptions are (although I need to figure this out still more for myself - I’m at least aware of having a blind spot here)
Solomonoff Induction and universal inference

Would improve12

Across all the days (not Zach’s fault!), I am finding the chronic enforced togetherness trying, by which I mean both the large number of bodies crowded into the room and the effort to follow the same train of thought at the same pace as a large group of more vocal people. I crave more independent time and space.
Probably the content a bit more structured, more specific exercises, less reading on our own
Maybe reading/discussions of the reading material could've been done in smaller groups again, like on the first day. Although perhaps the material is less well-suited for this due to being more technical.
More assertiveness over people is definitely needed, Julian was a good model for this who would more strongly elicit people to say where they were up to and stuff
Either just give intuitions, or make the maths very rigorous. The middle path was not great, the exercises were quite confusing due to the mixed and confusing terminology, and the lack of prior mathematical rigor.
more defined pedagogical goals going into the sessions, and scaffolding for this content. restricting to either conceptually overview of key results or narrowing the scope would be useful. breaking readings into chunks was not very effective when total understanding in a limited time is not possible and the main goals were not clear.
Maybe this will come later, but I think it would have been useful background context to see up front what people currently doing learning theory in AI are up to and whether/how this is useful
Some of the material might have been presented in a lecture, highlighting the most important findings - but ideally with a different lecturer. For my personal tastes, Zach was a bit too disorganized and confusing with explanations of what should be precise concepts.
Maybe have slightly more structured discussions of the readings
1. Randomly assigned pair of 1-1 Q&A for 10 minutes after every 1.5 or 3 hours of content or a breakout group of 4-5 people and a TA to facilitate any Q&A. We desparately need, need this please. Reasons: - if we have 1 big group and 1 instructor, if one person asks questions everyone else queue. - Also, some are just not comfortable asking questions when there are many people. that's why you can notice there's almost a consistent group of people who keep asking questions and the rest who don't speak much. I'd want to hear more perspectives/questions from the rest of the people too, and want more psychologically encouraged to engage and ask more questions. I'd further argue that if things keep going this way, there'll be certain tension and conglomeration where people more active in class will form clusters, and the rest form cluster. Not good for class synergy. 2. Release the materials 1 day earlier, please. - For example, I'd expect to have materials for Day N + 2 on Day N. - Why: the materials are dense, we barely had any pre-requisite reading focusing on each of these concepts. Meaning, unless someone has stumbled upon this before, it's their first time reading/learning the concept. Since we have quite a big variety of backgrounds, I find it highly infeasible this occurs for the majority of people. Therefore, the least you can do is to give people the slides content, or more simple sufficient, having the instructor gives out bullet points of concepts covered that day. We don't need exact reading list or slides if that's helpful. We want to gauge the main concepts, so we can prepare better and come armed with more context, and engage more meaningfully with the materials. This also flatten out the bias: making it more fair for all class participants and neutralize the effect of having seen the materials before leading to better performance in class. Otherwise, the graduates offered to the post-residency fellowship would skew to almost all people who have already had prior exposure to the content, or pure genius who can picks up on learning immediately on the spot. Do you think this would be unfair for people who have a different background, different learning method, and different knowledge base about these, but who are equivalently ambitious, hard-working, and can perform better if pre-context and independent preparation for the materials ahead of class? the point of the class should be that students have the chance to meaningfully ask sharper questions and have it resolved, or meaningfully discuss wth fellows. If I'm just learning everything for the first time at class then I'd just prepare to do it on my own at home, I think I'd learn better that way. If you feel meh and negligent about the first two points I shared above, and would prefer letting the class be as it is -- why are you spending all this money funding for us to come here and learn if not taking into feedbacks and try experimenting? after all, we still have 3 weeks and the materials are only gonna get harder. 3. The room is too hot
I wasn’t a huge fan of the reading for Legg’s thesis - I think it wasn’t precise enough on a few points I’d care about a lot (e.g. what was a probability distribution or not, exactly what was included or not), so it lost time getting intuition for this where trying to figure out the definitions was not clear. Perhaps a handout sheet to state just the definitions plainly would be helpful, particularly for then using these for the exercises
The presentation of the material could have been better, the presenter is clearly extremely knowledgeable but didn’t seem to be so on top of this particular material.

Other thoughts1

Probably still less conversation in the class about questions or comments not directly clarifying the content would be preferable. I think questions of where people are at in reading maybe should be done more explicitly to fully check on positions (where this is certainly my fault to an extent for not answering)

Reinforcement Learning 1 run 15 responses, 37 written

for pasting into Claude

April 2026 Tue 14 Apr David Quarel 15 responses energy 3.8/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.2 of 5

Teaching and format 4.1 of 5

Difficulty match 2.5 of 5

Energy 3.8 of 5

Read what they wrote37

Worked well12

The exercises were well-structured and informative. I also liked to have sufficient time to work on our own.
The exercises (I did the practical/coding route) were instructive and fun.
Again, I'm very grateful to have the cache of materials to return to later.
Working with Aiden
Very well rehearsed speedrun lecture, not getting caught up on inane questions, coding/practical exercises - im sure the proofs are useful but actually playing with parameters and debugging misunderstandings/pair discussion was the most useful
The presentation was a very useful refresher on tabular RL
Enjoyed the lecture, really liked the exercises (completed theory, started empirical)
Interesting exercises
The colab notebook was excellent (relevent info was reviewed, problems were clear + useful)
the lecturing content covered the exercise content well, all of it gave useful anchor for what the math track illustrated and the math wasn't too divergent from the slides.
The lecturer was great at explaining complex topics in clear and easy to understand ways. Also the format of having one 45-min lecture and then the whole day to do exercises with the choice of tracks worked perfectly as I could choose what I was most interested in and could work on the exercises (in a pair) at my own pace and speed of understanding.
Dynamic and efficient presentation to start the day !

What they took away12

I think I got a better understanding of why RL convergence results hold, and how this depends on e.g. the learning rate and the policy learning algorithm.
How basic RL algorithms work, with some intuition for the parameters involved and how to improve them.
I had seen much of the content before (although approaching familiar things with "the beginner's mind" is a very wise thing to do, in ~Buddhist philosophy!), so having the latitude to run variations on the experiments in the colab (instead of just straight implementations of the instructions) was welcome.
The difference between Q learning and Value Learning
scheme for optimizing agent decision-making and how to implement using gymnasium
That we don’t have a nice elegant expression for a stopping criterion for Q value functions as we do for value functions (only PAC bounds).
Understanding of Q-learning for RL foundations
Q-learning vs traditional RL
I think the fact that the Bellman operator being a contraction map (which seemed key in many of the derivations in the exercises) was a good fact.
The generality of the conditions on which the theory is built (esp. thm. 6.1 in the exercises) was pretty astonishing, exposing the wide range of tasks with potential RL applications as solutions. Hope to learn more about the limits to it, though
Just by changing epsilon, essentially one parameter, we can design solutions to "be more careful".
maybe temporal difference / eligibility traces

Would improve11

Maybe we could have had also briefly covered some other RL techniques, such as the ones used for LLMs. We did cover this already somewhat during the MechInterp module, but it would be nice to revisit now that we have more background on RL.
The initial lecture was quite fast. Inserting another short lecture/discussion right after lunch might have been good.
I think the biggest obstacle to my progress/enjoyment was my own exhaustion/saturation, after a different topic every day for the last while. That might or might not be something for Iliad to improve somehow (see below), but it certainly isn't David's fault! I normally find it difficult when the lecturer speaks very fast by default and makes it the listener's job to cry out for a halt when they can't bear it, instead of proceeding gently by default and reading the room, but I had no trouble following David. He came across as incisive, with delicious bursts of humour in what could have been a dry, stale rerun of old material (old historically relative to sota, and old to David as a learning human).
Idk
better environment than upstairs (initially, we moved downstairs later) for pair programming/discussion
Spend a bit more time on the initial lecture before separating into theory/empirical.
The exercises were too long to fully make it through in one day and in the end I think some mix between the theory and practical might have been a better mix. Equally, we could also have just assessed that for ourselves.
A little more lecture or q and a
Every once in a while is fine, but I wouldn't want one long block of exercises for every afternoon/session
During the exercises most if not all people seemed to have questions that arose at specific points of the exercises and I believe the answers to those brought some conceptual clarity to the teaching content, without specifically pointing out where. I think taking time at the end or at regular intervals to gather for answering questions, and then recapping the course material could help deepen the knowledge obtained this way. (I had the opportunity to listen half-attentively to the answers and that felt helpful, anyway.)
I would have liked to see some live coding for some of the answers of the notebook to see which parts or edge cases we could have missed when filling our solution in and maybe have a description of the intuition to draw to implement the equations into the code.

Other thoughts2

Could u order Chinese food for next weekend pls
I was on the math track; I didn't fill these forms up to this point, favoring some freeform personal feedback notes (to be shared), starting now I'll make an effort to write in both!

Reward Learning Theory 1 run 9 responses, 24 written

for pasting into Claude

April 2026 Wed 29 Apr Leon Lang 9 responses energy 3.6/5

Called “Outer Alignment and Reward Learning Theory” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.0 of 5

Teaching and format 3.9 of 5

Difficulty match 3.0 of 5

Energy 3.6 of 5

Read what they wrote24

Worked well9

The topics were mostly new to me but they were on my radar and felt like i was missing these so the selection was suitable to me (except for Richard's).
Even though there were quite a few lectures, they were varied and spaced enough for this not to be a problem in my experience.
Good clear presentation at the beginning, scaffolding on exercises
Joar’s lecture was really interesting; I appreciate having 2 guest lecturers covering different spaces and perspectives, very interesting to compare their representation of goal and behavior, reward and policy differently. Great choice of speakers.
It was nice to have multiple lecturers to get different points of views on the field.
Morning lecture and exercises (especially the first one) were great.
I enjoyed the varied format of teaching.
Guest lectures were very very good Exercises were good but a bit easy relative to the amount of work they took
Leon might be my favourite lecturer of the programme. He defaults to a slow, inclusive pace (without needing to be flagged down and begged, which can feel embarrassing), he doesn’t charge shame tax on questions (while also not letting them derail his arc), he laughs a lot like the Dalai Lama does, he knows how to choose specific, clear examples. I hope I “grow up” to have a lecture persona like his someday. I hope he does more of the teaching himself next time. Joar’s content was very cool. I wish we could get him to do a full day.

What they took away9

everything about IRL
It was interesting to learn about Joar's RL framework, and how it can be used to formalize reward function robustness and misspecification.
Joar's talk on STARC metrics and observation dependence for feedback
I'm very interested in the research direction of Joar's.
The concept of inverse reinforcement learning was really interesting, in the last talk.
To explore flexibility in reward models.
That RLHF needs very strong assumptions to work
Maybe this idea from Richard about how groups of agents combat discounting
Partial observability is really bad for RLHF. Also, reward optimisation is our only method for directing AI systems not anchored to or limited by current human preferences.

Would improve6

I didn't see the relevance and utility of Richard's talk, and to my understanding a lot of informal assumptions and handwavy explanations went into it, which made it seem professionally lower quality than others. The strong leaning on the utilitarian approach was rather alien and hard to digest for me and I realized that I'm not gaining value from it so I left halfway through, so this opinion might not be representative.
The computations in the first exercise were a bit laborious at times. Maybe the number of computations could have been reduced while preserving the educational value (e.g. by removing a state).
A bit scattered today between talks, exercises could have been a little more carefully constructed/concise
I would give some papers of the speakers to read before to be familiar with their research and have some more insightful questions to ask them, generally I found their presentations really engaging.
The material was not quite enough for a full day. I also found the first afternoon talk on agents and decisions distracting - too philosophical for my personal taste, with no concrete implementation. (One can philosophize endlessly - better for dinner conversation than course material.) On the other hand I’ve been enjoying learning from other program participants and trying to develop my own research plan on days like this with lots of break time.
Richard’s talk was less relevant and helpful. I think we’re all struggling a bit to juggle 40hrs/week in your classroom with the demands of our lives outside: I wish the schedule were divided into core/mandatory parts which build cumulatively on each other and optional/elective parts which don’t. That way, if we need to miss part (to take a job interview call or push for a deadline or similar), we could plan to miss an optional part and know we weren’t doing much damage to our experience of the month.

Singular Learning Theory 6 runs 164 responses, 240 written

for pasting into Claude

October 2026 Thu 15 Oct Brianna Grado-White 35 responses energy 3.8/5

Rated against what each person expected. 0 is as expected.

Overall +0.34 better than expected

Content +0.28 better than expected

Clarity -0.38 below expected

Motivation -0.25 below expected

Exercises -0.03 as expected

Readings +0.16 as expected

Other activities -0.05 as expected

Pace +0.44 too fast

Difficulty +0.69 too hard

Novelty +0.75 mostly new

Read what they wrote40

Worked well11

I think walking through concrete examples applying SLT was helpful in getting the concepts to connect
This working lectures were clear, and quote advanced
I really enjoyed Lin’s talk, I think it was the most mathematical-audience oriented talk so far and catered well to my knowledge and familiarity. Lin gave some of the best explanations of grokking, double descent, and the staircase I have seen so far. He also had very good suggestions for questions to pursue (in fact I wrote up one of them for my project proposal!)
The slides were good, once I really understood a bit of algebraic geometry etc.
Shaowei's talk.
i really appreciated the first speaker's presentation. it was more mathematically involved than i could follow at times, but i really appreciated the clarity and structure, and how we went from simple statistical models to algebraic geometry and back
the exercises were excellent. the content was awesome. the discussions with colleagues and the lecturers were great.
lectures were thorough, clear, and interesting.
The lecture slides were great, perfect balance between math and visual/organized info.
lecture quality and clarity
The exercises were good

Would improve21

I think today would have been a bit easier to follow had the broad motivations been presented earlier and upfront. Instead I think it followed a route of introducing things as they were developed which was difficult given so many levels of abstraction and new concepts.
I thought both lecturers were great, and the subject matter was very interesting. This was one of my favourite days so far. I was very impressed and I think I learned a lot from them.
I really really enjoyed today! The one comment I would leave which is very specific to today is perhaps having a brief interlude where the connection between the morning talk by Lin and afternoon talk by Dmitry is explained! They were both using the word singular models but meant actually very different things by it. In fact in the morning Lin said most models are singular but by afternoon Dmitry said most models are regular!
As a non-math person, I found the contents a bit heavy and difficult to parse and used Chatgpt extensively to understand. I would really exactly one thing of the lectures - If we could have the slides beforehand, that would be idea. The people not familiar with the background can look up and try to make sense of the pre-requisites.
The Slides for the second Part we're Not ideal, maybe a more rigorous writeup of everything (both First and second Part) i.e. that goes into more depth and is less handwavey (or pointers to papers that are rigorous)
Catalogize materials in terms of prerequisites maybe? I don't think my path to impact lies in becomes a physicist and then applying phisics to ML. But then I struggle to motivate myself to work through this material and I don't know when it's worth paying more attention. (I've already wasted years studying math I never used and forgot.)
The post lunch lectures could be a bit clearer.
Better slides in the second half.
I think today's lecture might have been improved by breaking them all in half and having mini-exercise sessions in between all of them. There was a lot of theory here and it was easy to get lost; having breaks to catch up and do exercises more regularly might have helped.
The lectures are good. Would be nicer if we can learn a bit more about the recent literature on singularity theory and the open questions that people are interested in.
Prepare the slides better (Dmitry). First speaker was excellent.
i think having two speakers, one presenting big picture and one working things out in more detail is a great idea, but the second speaker's lecture was really hard to follow in the sense it was not clear to me what was assumed and what was shown and in general where we were going overall, i think it would have been helpful to spend some time motivating why we should care about SLT
Maybe the second half can be better structured
both lecturers had great insights to share. however, Dimitri could have improved the preparation, and polished his slides more, for a more efficient impact on me.
The first part of the day was engaging. But the second half didn't really connect with me. Also, both sessions were lecture heavy. Would be better if there is a balance with exercises and coding.
nothing.
The last lecture was hard to follow from the slides.
Not much - I really enjoyed this one.
both talks were good!
Maybe some more preparation of the presentation for the second half.
The second part of the day, while it was very very interesting, was a little bit hard to follow. Maybe, for the second part of the day, a possible improvement would be to concentrate on one specific result and focus a bit more on the implications for safety.

Changed my mind7

Loss isn't always the best measure of how a network is developing and generalization is an important thing to measure for capability development.
I think today convinced me that my particular skillset (the kind of math I’m familiar with and do) can be used to understand key questions in ml theory (grokking for instance!). I’ve walked away from this talk seeing good directions in AI Safety I can apply my knowledge to!
Mostly new, so no.
SLT seems less impactful, after today.
got more motivation into research of the mixture in both theory and practice for AI safety and MechInterp.
I was told that SLT was substantially algebro-geometric, but such seems like an overstatement. Nevertheless, there seem to be some cool problems to solve related to the content from today.
I've since become very motivated to study SLT, and am considering researching it in the context of math grad school or upskilling in the research engineering side, or both.

Other thoughts1

I was a bit low energy, so I might not have the usual level of energy to follow and understand.

September 2026 Thu 17 Sep Kai Ogden 37 responses energy 4.0/5

Rated against what each person expected. 0 is as expected.

Overall -0.06 as expected

Content +0.17 as expected

Clarity +0.39 better than expected

Motivation 0.00 as expected

Exercises +0.28 better than expected

Readings -0.38 below expected

Other activities -0.18 as expected

Pace -0.83 too slow

Difficulty -0.78 too easy

Novelty +1.08 mostly new

Read what they wrote58

Worked well19

Excellent pedagogy and big picture motivation.
I liked that it was on a whiteboard, it made it much easier to follow. The teachers could explain things really understandably. I liked that they worked through some toy examples on the board. I also really respected their epistemic humility and it helped create a great atmosphere for asking questions, admitting confusion etc.
I think the short lecture followed by exercises system was quite good, but everything was a bit too slow - could have done several of these blocks probably rather than just three.
I liked the pace and especially the whiteboard lecturing. I follow better when new concepts are thought using a whiteboard vs slides.
The exercises well well chosen.
I really liked the bite-sized sections of lecture+exercises!
the long discussion at the start of today about how the theory today fits within ai safety, what the hopes are how it could help etc before even going into the theory details. every lecturer should do this!!
The whiteboard -- I did like the format a lot, reminded me of Uni lectures and the "How to Speak" lecture by Patrick Winston https://ocw.mit.edu/courses/res-tll-005-how-to-speak-january-iap-2018/ where he claims that comprehension speed is about the pace of writing on the board. Previous lectures weren't as easy to follow because of slides, while here you can write along and digest more slowly. I also liked that the motivation for devoting so much time to singularities specifically was given. The lecturers today had a vision that most of the time should be spent on singularities and implemented it. The exercises were also quite fun, I enjoyed them a lot. As well as the format, aka lecture-exercise-lecture-exercise.
The willingness of the lecturers to help during the exercises to make sure nobody would fall behind
i kind of liked the blackboard lecture style a lot (compared to a presentation with slides)
Recap of exercise 3.4 on the board was very useful to get insight.
my favorite day so far, the flow between teaching, exercises is quite good with the teaching sessions not too long and well designed exercises to gain some insights to the material hands on.
Discussion of the solution on whiteboard
I think the lecturers were good and engaged well.
Geometric intuition for local learning coefficient.
Short lecture, dive into exercises
Discussing with the lecturers in the off time clarified most of my questions , they were very helpful!
Kai is fantastic. He's an excellent communicator and clearly has some mastery of the topic. Matthew as well was very easy to ask questions.
Whiteboard is the best. Makes the pace perfect.

Would improve27

I think the class was pitched at the wrong level. Too much time was spent on trivial things like curve sketching and computing derivatives. On the whole, I feel that very few concepts were covered today. This was very pedagogical and I was able to follow everything. However, the level of this class was more appropriate to an undergraduate class rather than a class of PhDs and postdocs.
The intro & motivation was a bit too long. I think the pace could have been a bit faster, and this would have allowed us to cover more material.
I think the first two lectures were slow in terms of delivery of new concepts, and I think it would have made sense to have slides so that walking through the examples more seamless than it ended up being. The content was interesting overall and some of the exercises were good.
Some of the explanations at the whiteboard were not very clear and some others were a bit too elemental. At some points I think that the questions of the participants were not really relevant to the explanation, so I think the speaker should not be afraid to delay the answer and continue with what he wants to explain.
The initial discussion was great. However, the lecturer wasn't comfortable with a lot of the maths involved- which is in principle okay (maybe less okay in something like SLT though). The problem was that people kept asking irrelevant technical questions about some of the constructions. A lecturer with more mathematical maturity maybe would have cut off these discussions (they were mostly useless and pedantic) and would've continued with the intuitive explanations he was giving. Despite this, the materials were really good!
I think the day wasn't dense enough in material -- most of the time we just practiced differentiation and multiplying jacobians by fixed vectors. It's fun, but not for too long. Increasing the density of material and covering certain corner cases, where things break overall, could be valuable no matter what the second day covers. Adding examples of models from standard statistical courses and discussing where we could break them for SLT and what does it allow us to analyze could be interesting. E.g. showing some key non-identifiability results + LLC for these models as a comparison. If for them, unlike neural networks we discuss later in the course, things start to break, that justifies an additional discussion.
I think the pace was rather slow and would have liked to see more of SLT or more technicalities. We looked at examples of losses that were fine theoretically, but would have liked to see some of this formalism applied to real networks, even though just with approximation.
I was happy with how today went.
Lectures only covered a general overview, and hinged on too simple cases. The exercises were useful, but it would be useful to see some deeper and technical results during lectures as well.
The discussion at the start of the day was nice, I think having more time to discuss these broader questions would be nice, not necessarily on this day. There seemed to have been a misunderstanding in what the point of the lectures was, as I believe they were meant to provide a schematic overview with exercises going into the more specific details. Many participants seem to have expected more formal details during the lecture, and clarifying this disctinction at the start would probably have made everyone accept some level of glossing over details more than they did. This time could for example be spent on going into more detail on the particularly complicated stuff, like the volume scaling law. I think the content of the first block about singularities could've been condensed more, as most people are familiar with the underlying maths.
Add the usage of https://www.geogebra.org/calculator or similar (say contour plots) when we want to discuss surfaces and landscapes for the ease of visualization of valleys etc. The day started with the oral motivation for SLT (no math), and I found it not too easy to follow given that there were no slides. I think that part would benefit from slides. Also, I hope it's gonna be addressed on Day 2, but if not, I'd say that an "empowering example" or a success of SLT to be shown would be a nice point to start. Actually, applies for every topic we cover at the intensive. The lecture would've benefitted from more preparation of the whiteboard lecturing itself, as Matthew was sometimes unsure of what he was going to show, or at least that was an impression. Also, I am against any lecturing right before lunch -- too intense for hungry brains. The whiteboard would've benefitted from elevation, as writing on the lower part was not visible to the participants in the back. Also hope it would be addressed tomorrow, but showing how specifically statistical learning theory fails that we need SLT would be very helpful and motivating.
Having short slides on top of the drawings on the board during the oral presentations to help the audience following
- I felt like the talks often lacked structure, which contributed for the unnecessarily slow pace; also, more content would be nice - some exercises were too much undergraduate Calculus-like and the "insight density that they offered" was small - also, I appreciate the presentation + exercise structure of the notes but feel like given the small extent of the presentation part, it would help to clearly separate which results I should retain from something I learn from an exercise which has a particular context; (also maybe some plots would be a nice addition)
Generally was good. However, SLT seems to be a combination of (1) conceptual insights into loss landscapes and learning dynamics and (2) rather technical mathematical results and conditions that are useful for understanding, deriving, and evaluating the conceptual insights, but aren't necessarily otherwise important. It might've been nice for the teaching to distinguish more clearly between these. Which parts are technical waypoints and which are the big ideas?
I had quite low expectations, but overall the material was quite clear. It is, unfortunately, mostly undergrad calculus and algebra, hopefully the next day will give better perspective on things.
Most exercises were warm ups, very quick to finish and far from realistic settings. The lectures were not as substansive as I would have hoped. A lot of discussion that was tangent to the goal of the day. The lectures felt a bit unprepared. Definitely use varied pedagogical supports (slides + white board) to improve teaching. Too much hand waving without visual support to structure and ground the arguments
honestly today
Visual aids, slide wth the visualisations of the graph
I think some of the material could be better motivated, and also some pre-reqs could be assumed
Pacing on resolution of singularities.
The beginning felt too easy and the first exercises also easy. I would have liked more motivation and more difficulty.
I would make the material a bit higher level. Possibly also the connection to statistical physics could be highlighted more. Motivation/introduction went a bit overtime.
First half of the day with Mathew was fine but it didn't seem as prepared.
We should have more coding exercises.
Today was a really nice day. I loved that it was at the whiteboard, was great for setting the pace.
I think some of the mathematical formalism could have been made a bit more rigorous for clarity, although I think the overall ideas were intuitively conveyed. The pace was also a bit too slow and wecovered relatively little content vs. most of the time on exercises.
I found the pace too slow and too easily disrailed by questions that were not really important. Overall it was set at a level not appropriate for the course. The material was in principle new, but it over emphasied low level concepts and progressed very slowly for someone with 'mathematical maturity'. Some of the digressions were brought about by participants

Changed my mind9

SLT looks like the String Theory -- lots of math, some captivating some complicated, but with no practical utility. I have asked Matthew whether SLT made one apriori prediction and if it was successful and he told me he knows none. He did say though that SLT helped him see the things differently and in a richer manner -- more emphasis on that should've been made.
SLT is interesting, i need to learn more calculus
Nothing, we did not get very far into SLT.
SLT is very cool and i want to know more about it
n/a
Degeneracies are features, not bugs, in learning.
Perhaps SLT may be useful
I discussed with other participants about EA Global New York. I learned how algebraic geometry can be useful for AI safety.
nop

Other thoughts3

The schedule format in the google doc works well.
n/a
Excited for day 2.

September 2026 Fri 18 Sep Kai Ogden 37 responses energy 3.4/5

Rated against what each person expected. 0 is as expected.

Overall 0.00 as expected

Content +0.07 as expected

Clarity +0.07 as expected

Motivation -0.03 as expected

Exercises +0.13 as expected

Readings +0.11 as expected

Other activities -0.31 below expected

Pace -0.37 too slow

Difficulty -0.13 about right

Novelty +1.03 mostly new

Read what they wrote40

Worked well9

The online presentation we got was quite good. It put a lot of the material into perspective.
I really like having two days on the same topic. Really helped with retention.
- Having two days on it, gave time for things to settle - Grounded in actual papers (Watanabe, the LLC estimation papers)
I think the discussion at the end was interesting - a lot of people asked good questions. I also had the doubt about how eliminating data is much easier to treat in Bayesian learning than SGD learning.
Kai is a fantastic teacher, Mathew was also good but felt a slight bit unprepared at times.
Lectures were clear.
Connecting LLC drops to phase transitions in models.
The online lecture -- it finally provided the megapromise for SLT showing that SLT predicts stuff that the actual learning ends up doing. This belongs to the morning of Day 1, like this should be the starting point. The lecture content was great as well, but for the improvement suggestions see above (such as should've been twice shorter). The exercises do not disappoint, they were nice, I liked them. The structure -- I think I should stop praising lecture-exercises-lecture-exercises because I have been doing that, so I'd just say keep it up. The essential points of maintaining "no lecturing before lunch" and "lecture under an hour, better 30 mins perhaps", as it was today (30 mins was great) were observed too. And I liked that exercises were given with suggested extensions, such that you could either do just the 3 core ones, or all 7. Keep that structure up.
exercises on phase transitions were nice because they were simple.

Would improve21

I would have loved to hear more about open problems and research directions. This is something that occurred to me and also applies to the other topics and lectures.
I would add coding demos for exercises.
- Some of the notation was really unfamiliar and not introduced much, a notation sheet or sticking closer to day 1 / Watanabe's notation would help a lot - Making the LLC estimation exercise part of the main day, to see the theory in practice
Too many graphs on the slide, better to have less and explore in detail. Write a bit bigger on whiteboard.
The remote contribution to the lecture did not work as well as the in person part. The material was more advanced than last time, which was very much appreciated.
I think there is probably too much focus on the mathematics in the lectures, which has generally been quite easy and posed no issue in exercises. I've found it significantly more difficult to get enough conceptual understanding, and I think the lectures should prioritise this much more. I think this debt made following the last lecture - which seemed very interesting! - quite punishing. I think many of the exercises were not particularly insightful as they felt like simple algebraic manipulations or computations in multivariable calculus. Exceptions included the exercises on why d/2 and not d and Ex. 4.2.
More context about how this relates to AI safety.
The pacing was a bit slow. Too much time was spent on basic mathematics and a few important results were just briefly sketched.
In fact, the second day of SLT was much better and more engaging in comparison to the first one. Potentially, some additional exercises on computations of degeneracies -- at least in common statistical models and explicitly seeing their implications could be really fun.
An almost 2h remote presentation, followed by discussion, with no scheduled breaks, at the end of the day was quite heavy. Great content otherwise!
It was good to get the applications of yesterday's basics. I believe some of this could be joined into the previous day to strectch the buildup of theory shorter.
I think the material should be given in a different order so that the main theory is presented (rather than elucidated in the exercised and focusing on basic definitions in the lecture). At points certain knowledge could have safely been assumed (e.g. basics of Bayesian statistics)
More practical code examples for estimating lambda.
feels like not enough new material for an entire day and the lecture times today were a bit too long to be effective
Regarding yesterday I said: "- I felt like the talks often lacked structure, which contributed for the unnecessarily slow pace; also, more content would be nice - some (not all) of the exercises were too much undergraduate Calculus-like and the "insight density that they offered" was small - also, I appreciate the presentation + exercise structure of the notes but feel like given the small extent of the presentation part, it would help to clearly separate which results I should retain from something I learn from an exercise which has a particular context; (also maybe some plots would be a nice addition)" These things still apply to today. I'll just add that the talk by Miguel was too long and the slides had too much information which made it difficult to follow. (I would have liked to hear about the papers he mentioned in a more comprehensive manner). Also, it would be nice if some of the things he presented were the integrated as a continuation of the main talk/notes material.
Adding a group activity would be cool I think
Exercises were useful, but the ratio of symbol manipulation to conceptual insight wasn't great. I think it might be possible to construct more examples that make things more visual/build more intuition. The phase transition problem was pretty good, for example. I also spent some time playing around with posteriors and V(epsilon) functions in Desmos and found it quite useful.
I don't feel that I obtained much insight from solving the exercises. I found the talk at the end a bit too dense. I found it very convenient to have written detailed notes. I think one of yesterday's exercises 2.16 was a bit confusing. It is given an example of a reparametrisation that introduces degeneracy on loss landscape. If I understand correctly, this can be made more precise in a general setting. Namely, at a critical point, reparametrization by a diffeomorphism must preserve the Hessian as a bilinear form. In the example the parametrization is not a diffeo at the origin, so it is expected that the signature of the Hessian is not preserved.
More preparation before the lecture suggested for lecturers: say running it between themselves before running it in front of the participants. This suggestion comes from the fact that sometimes Matthew was lost in what he was saying, but I can't recall his specific phrase. Smth along the lines of "Okay this actually does not belong here." I think if this is done he'll be great! For the online lecture, the soundcheck and other stuff better be run before to avoid fixing problems during the allocated timeslot. Also the lecture itself was quite long (2 hours) which inevitably leads to losing focus without a break. If we don't wanna be doing a break, perhaps a Veritasium approach (two plot lines, the movie aka lecture switching between the two to keep engagement, the trick Derek borrowed from cinematography school) would help. In fact, lots of cool stuff was in the end, where the concentration was already lost, so perhaps the talk could've been twice as short without the need to go into details early on. I'd say Julian was quite a master in this: he had like 160 slides in his MechInterp day, but it actually felt extremely engaging and not tiring at all, as each paper had one single figure from it on the slide and was covered quite briefly, with a single idea behind it, illustrated with that figure. Quite an exemplary approach.
The lectures need a bit more structure (dual support using slides is very pedagogical). I feel like these two days could have been condensed in about 1.5 days, giving time to read and discuss papers in the last hours of each day. This way, Miguel's lecture would have hit the target better and conveyed more meaningful questions and discussions.
There should be no remote presentations during the intensive. Pace can be a bit faster.

Changed my mind7

SLT has quite a few compelling ideas and it seems like it could use a lot more active researchers! But I'm still not sure how clear the link to AI safety is. Not just in terms of differential advancement of safety - I'm not sure what the theory of impact is for positive differential impact at all.
Not particularly
n/a
SLT provides a rigorous foundation for developmental stages.
became a bit less sure about if im convinced by SLT, but will spend some time doing more reading to make sure!
I realized that maybe chasing the strict minimum of the loss function landscape is not really needed. Maybe the landscapes is the wrong thing to study. The secret to generalization might be hiding in a different thing altogether (the early stopping generalization had a point, just like other methods like random deactivations). Btw the demo that, unlike GD/SGD, Adam and Muon first learn smaller components, which might be "noise," was quite surprising and exciting.
SLT during training could be interesting but needs improvement in LLC estimators.

Other thoughts3

Today and yesterday was very tough because my calculus is rusty. As a computer scientist I’m much more familiar and comfortable with discrete math
n/a
Outstanding two-day deep dive.

August 2026 Tue 11 Aug Zach Furman 31 responses energy 3.9/5

Rated against what each person expected. 0 is as expected.

Overall -0.10 as expected

Content +0.48 better than expected

Clarity -0.10 as expected

Motivation -0.19 as expected

Exercises +0.52 better than expected

Readings +0.75 better than expected

Other activities -0.50 below expected

Pace -0.03 about right

Difficulty +0.39 too hard

Novelty +1.13 mostly new

Read what they wrote59

Worked well15

The exercises and technical aspects in the talk were nicely explained
The exercises were amazing way to actually reinforce understanding of the topic
Loved the handson exercise solving
Desmos graphs were great!
Zach's high level break down of all concepts without scaring people off with heavy duty AG lingo.
Exercises helped a lot.
the lecture!
I think doing exercises was important, albeit unpleasant at times.
Doing exercises to clarify concepts
First half of the slides
slides were a good intro. many problems were doable at least until the end, the tutorial as a whole is likely a good resource if we go start to finish rather than skip around
- The notes are amazing
Discussing the exercises with peers.
The morning lecture at the beginning was useful. I also appreciate that Zach talk about his own interpretation vs some consensus in the community.
The excercises forcing us to put what we learnt into practice was very useful, it's nice to go home feeling like I've actually pressure tested my understanding of it

Would improve23

I would have liked more references to follow up on some of the math intuitions.
Just one sentence on big picture regarding ai safety!
Nothing really.
I’d like more time to process and work through the exercises as well as perhaps more time going in depth. I realize that this reads like two things pulling in opposite directions- I guess I was most interested in SLT coming in and would’ve liked to spend more time on it! But definitely came away with a clearer understanding and a vague idea of where to go from here!
We talked about it in the end but it felt unmotivated, partially due to scehduling. But I feel that I understand today but am not compelled by its application to AI safety.
I would make it a bit clearer exactly what exercise the class is working on currently, especially since you are encouraged to go and work outside for a while (I also missed some 'lecturing' parts because I didn't know we had been summoned back in).
Slt could have two days dedicated to it with lots of exercises to motivate the material. Also, showing the connection to safety would be useful.
During the morning session, the questions round extended too much in time and it was hard to engage.
Maybe spending a bit more time explaining how SLT fits into AI safety (there's a chance this was included and I missed it) Nit-picky, but maybe alternating more often between lecture and exercises
Lecturers spent to much time answering irrelevant/tangential questions. In general, I think the presentation could have been *half* as long as it actually ended up being if people stopped asking all types of irrelevant questions. I don't blame people for being curious, but I really would wish that lecturers more often say "you can google/claude this" or "come to me during the break". I also think the presentation was not that good and it was not always easy to follow (from the exercises/texts things were easier).
Maybe going over some exercises together early on, as some of the definitions felt different to how we compute them in practice. I think it was great to have solutions, but would have been helpful to maybe work in groups? I worked with a buddy, but talking to other people it seemed they felt like they were the only ones struggling. Also the timing for the exercises seemed fast (as in, too little time to work) but once we see the relevant trick in the solution, the short duration made more sense...not sure how to improve this hahaha, maybe
When explaining what happens in the bayes case, you could make it very clear on the slides what the type of each object is, and maybe also explain what the bayes setup is. Also maybe explaining/excluding susceptibilities.
Sometimes the lecture explanation felt rushed and confusing, eg some equations or plots shown without properly defining variables etc. Also connection to safety a bit unclear
Let speakers know that they can skip answering questions for the sake of time. We should have done the mysteries of deep learning today. Feedback for Zach in particular: I got the feeling he kept getting nerd-sniped by the questions. I think the slides could be organized in a more focused way.
more organization around solving the problems. probably want to split into groups and do entire sections at a time rather than 5 mins for a somewhat involved problem. also if a problem requires extra knowledge (even simple such as Taylor expansion or knowledge about what invertible implies) may want to add it to the hint
- The written material is amazing, really well written. Nothing needs to change here I guess, except maybe making more materials like this for more complex stuff, but it does is super great to follow, exercises broken down into easy parts that allows you to construct the theory alone, super fun - The class took more time than it should and thus we had less exercises time, and did less exercises than what was initially planned. Also, we had to rush through a lot of them. In the first lecture of the day, people were doing 10 "curiosity" questions per slide (questions that are not like "I don't understand the material, please re explain" - which I'd say are fair and fine, but the questions were actually like "I saw this kind of behaviour how does this relates", which is also fine if not done in a way that actually prejudicates the schedule of the day). One easy way to improve would be to say for the professor to be more mindful (for ex., when prompted with a large "curiosity" question the professor could provide a 10second brief answer and say "ask me later" or something)
I would not do a session of timed exercises. Instead guided sessions or everyone works in their own pace.
A lot fewer questions during the lecture. I didn’t get much intuition out of the lecture, it would be nice with a concrete example. I did the exercises and often thought afterwards: why do I care about this? How does it relate to safety? Sitting inside the lecture room and doing exercises was uncomfortable and didn’t allow for discussion with peers. But when I left to do the exercises on my own I didn’t know when the lecturers would go through some material. Would be better if they let us work on the exercises in groups and gonged us in every 25 minutes or something.
For me, some basic concepts were explained a bit too exhaustive/lengthy, while being quite fast on the more interesting/involved topics; again, I would have preferred to spend less time answering (partly only tangentially related) questions, and stick more strictly to the planned curriculum — this way, we would not have needed to skip the more interesting advanced parts …
Talked to Miguel the TA and apparently many of the prerequisites and the build up got cut quite significantly. I'd prefer to do the Mysteries of Deep Learning first to build up the intuition before we go into the materials today. I also wish that we could be instructed in advance what sections and subsections to read prior and do the preparation of the reading prior to the class. The lecture in the morning where we talk about intuition was quite helpful. At this point, I'm still uncertain as to why and how degeneracy is relevant to generalization. And most importantly, I'm still unconvinced as to why SLT at all apart from that they are mathematically beautiful.
I think any course content should either be i) taught during the lectures as if we're completely unfamiliar with it, or ii) something we're explicitly told well in advance to learn before the lecture. The first hour or so of the morning session was great, but then suddenly it got way more technical and explicitly assumed learning theory background that I didn't have and I got completely lost. I was doing okay in the afternoon sessions, but then started to fall behind when the exercises started referencing content from Section 1 that we had skipped. At some point, once I realized we'd be largely going through the material independently during the session anyway, I decided to just proceed linearly from the beginning of Section 1 and ignore the instructor's pacing. Then during the break in the middle of the afternoon session I was having an interesting/productive conversation about AI alignment with some of the other attendees, so I opted to continue that conversation for another hour and a half rather than return to the session. I plan to return to the material this weekend, after the Mysteries of Deep Learning lecture.
The pace was a little bit irregular, and I would have appreciated spending more time on the free energy equation and deriving it, rather than all the other things which could have perhaps been given as initial readings.
honestly I thought today was pretty much fine. The lecture was really good and the excercises were quite helpful. If I had to change one thing it would probably be doing fewer, but harder excercises, rather than many short ones, however, I definitely felt like the questions forced you to use conceptual understanding rather than just turn the handle which was good.

Changed my mind15

Want to learn more about slt
Great to learn about deep theoretical frameworks to analyze deep learning models.
further validated my gut instinct that SLT is both beautiful mathematically and useful empirically
Thought that analyzing loss landscape was hopeless, but SLT changed my mind. It is a worthy attempt at generalization.
Learning Bayesian stuff might be worth it (which means a lot coming from an algebraist)
SLT seemed different from what I expected.
I didn't know what SLT was about and now I kind of do.
SLT is interesting. I heard people describe it as rather “esoteric”, but there are some nice concrete results
I am leaning more towards switching to a career in safety now.
SLT seems more approachable than before, was able to do more exercises than expected
Understood a little more details about SLT, great
SLT seems less clearly related to safety than I originally thought.
I'm uncertain on why SLT at all and why SLT is popular in alignment + safety space only but not elsewhere.
I was new on SLT, and the free energy equation and decomposition were very interesting.
Not really, I came into this sort of knowing that SLT was difficult to apply in practice to things like language models, and also that it helped explain some cool results, however it was very nice being able to finally take a good look at them

Other thoughts6

Again just a sentence or two on relation to safety. I can imagine but would like that confirmed or rejected
It would be nice to moderate questions from the participants somehow. It seems some people are trying to make themselves seen for the fellowship and the constant questions make it super hard to concentrate onto the actual lecture; which then tends to run late
We should limit the amount of questions per person
A lot of unnecessary and rather trivial questions were asked today, which made the thread quite difficult to follow. I also think the structure of the SLT talk could be improved. We didn’t manage to cover many of the important details, which left some gaps in understanding. This wasn’t because of the teacher, in fact, I think they did a great job, but mostly because the large volume of questions disrupted the flow and took up a significant amount of time. The exercises also didn’t help as much as they could have. I think it would have been more effective either to work through them in a guided way or to allow more time, rather than just 10 minutes per exercise.
If I were to get into SLT more, I'll probably be reading the grey and the green book first and catching up on the materials independently first then review the lecture.
N/A

June 2026 Wed 10 Jun Kai Ogden 12 responses energy 3.8/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.3 of 5

Teaching and format 4.3 of 5

Difficulty match 3.3 of 5

Energy 3.8 of 5

Read what they wrote11

In their own words10

The lecture notes are great, teachers accessible and explaining very well. I was slightly confused during the lecture but things clarified later during the day.
I think it worked well today. We had tight feedback from the teachers and we had timely breaks.
Interesting material. The opening talk could be improved, I don't think it conveyed the main ideas very clearly. The discussion leaders (TAs?) were very good! About the format, I felt rushed to red and do exercises at a particular rhythm. I don't necessarily see the point of sitting around reading some notes compiled by people who are in the room and could be exposing the material on a white board. This could also led to more interaction with participants. In general, I have a strong preference for whiteboard/blackboard talks as opposed to slide talks - whenever the formulae allow. I personally would have liked to spend more time on this topic. The idea of the learning coefficient is very interesting, and the expansion of the free energy formula deserved a longer treatment!
Content was super interesting and lecturers explained technical concepts well. I think the problems would have been better run in a group work setting (but that could have made the time crunch worse).
I very much enjoyed the technical contents of today's module. It felt a little bit rushed due to limited time. Personally, I'd have given more time to main contents of the lectures and spent less time on the introductory part. For the lectures, I must confess that the alternation between discussions and exercises piad off at the end of the day, and it would've been better withouth the rush. I would maybe use more the white/black board for the explanations, rather than displaying slides/documents in the TV (hard to see).
- The explanations were crystal clear - The most interesting for me was the connection with safety and the connection with empirical results - I think doing the reading under a constant timer was a bit stressful. It might have been better to have longer (40 min) sections instead of 15 min sections to reduce this constant time-pressure
nice to have concrete exercises, but only doing it alone felt more isolating than the previous days of discussions
Good format. Having lecturers in person today was great, and felt better than yesterday. Kai and Matthew were both extremely helpful and effectively conveyed their enthusiasm for SLT as a proposal for AI Safety. One criticism perhaps is that we covered quite a lot of material today- we literally tried to complete the full 60 page SLT notes by the authors.
I liked getting time to work on problems. I think it might have been helpful if the intro at the start of the day was much quicker, since I, and probably others have more energy early in the day, so that's the best time to do problems. There was a fair bit of discussion in between problem sessions and I think that plus some sort of lecture like Zach's at the end would suffice. Tldr; I think more details first, generalities later could have been a nicer workflow today.
Talk pacing and content were good, very well-prepared reading material, exercises varied from very simple to interesting

Other thoughts1

there is a typo in ex. 2.13 (b) - "any" appears twice (pg. 22 lecture notes)

April 2026 Sat 11 Apr 12 responses energy 3.3/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.3 of 5

Teaching and format 4.2 of 5

Difficulty match 3.3 of 5

Energy 3.3 of 5

Read what they wrote32

Worked well11

Notes were great, presentation was great and very glad we worked through exercises
It was nice to get some hands-on exercises
lecture notes were so good!
Matt and Kai were incredibly lucid and energetic and solicitous teachers; if l were in a semester-long course, I would be thrilled to discover they were the TAs.
Initial lecture, exercise and discussion format - all very good! (And this presentation is massively better than what was available online around 2 years ago)
The lecture notes provide a great overview and interesting exercises.
Self Study at my own pace
Exercises integrated with notes was great
Excellent day - great, organized mix of lectures, reading, problems.
Loved having multiple TA's
Mixture of lecture and exercises, reviewing the exercises/readings, minimal unsupervised learning, different instructors chiming in

What they took away12

The insight that the same model parametrised differently can have a different loss landscape geometry depending on degeneracy in the parameter-function map.
I now understand better how the posterior distribution shifts from higher-loss-but-lower-complexity to lower-loss-but-higher-complexity as the number of training samples grow
volume scaling definition
Links to sources to read later
Good coverage of the ‘singular’ definition and examples for this
The way in which the free energy can be used to explain the balance between complexity and fit (in the Bayesian setting).
How neural networks generalize; Internal Model selection.
There's predictable ways that parameters change and phase changes make them less predictable.
(I was only partially engaged, since I'm very familiar with this material, but was still useful to listen in on the discussions)
How all the different ingredients of SLT fit together, and what assumptions were being made in each place. Also, identifying how SLT may help (or not) in alignment research.
Just the format was great, having explainers, presentations and very useful exercises
Derivation of free energy and llc calculations

Would improve7

Perhaps tell participants not to necessarily work through exercises in detail given the time constraints but balance between working on the exercise and then looking it up/asking Claude
Maybe make more links with actual AI Safety stuff
It was too much too fast for me. At some point I stopped trying to follow along or do the exercises and reverted to reading the document from the beginning at my own pace. I wish I could have been open about this and gone out somewhere quieter.
Some exercises later in the day were a bit too easy, so I think there would be scope to push a bit deeper in some exercises, and to make the link when introducing probability distributions clearer. I think understanding the basic definitions deeply is quite central, and I have this to a reasonable extent, but suspect a bit more on this might be possible for complete clarity
1. I want to read counterargument or adjacent-varient theories to know how confident I should be about the likelihood of this theory and how strongly I should pursue it. 2. Implications of SLT to capability scaling or alignment: exactly how does SLT influence mechinterp/devinterp/eval practices? Exactly how do knowing this inform our understanding of current bottlenecks arguments? How popular is this theory outside of AI safety theory? How can we translate this theory to practice? 3. We spent 90% of our time on math proof, and 10% on overall lecture (which still is math-oriented); we would have more impact and solidify our knowledge more if more time is spent on the above points. As participants backgrounds and next goals are broad, spending an exorbitant amount of time on math may not be fruitful endeavor for everyone. Could we have dual-track: splitting groups into two streams; or just simply do 50/50 (50% on math and 50% on discussing the above points).
I would love to get some more useful intuitions and visualisations in the notes or on the slides!
Spend a little more time on the preliminary information first, but we came back to it later (this worked but could be more intuitive the other way around)

Other thoughts2

Overall great day! A lot of value
This felt like the best day in terms of organization and material. Even though there was a lot of new information, the pace, balance of exercises, reading and lecture, and breaks was very effective.

Steganography and Backdoors 5 runs 104 responses, 125 written

for pasting into Claude

October 2026 Mon 12 Oct Julian Schulz 33 responses energy 4.2/5

Rated against what each person expected. 0 is as expected.

Overall +0.22 as expected

Content +0.25 better than expected

Clarity -0.19 as expected

Motivation +0.34 better than expected

Exercises +0.06 as expected

Readings +0.05 as expected

Other activities +0.16 as expected

Pace +0.41 too fast

Difficulty +0.34 too hard

Novelty +1.06 mostly new

Read what they wrote39

Worked well12

The claude code exercises
coding exercises and workspace management
The exercises helped, must most of the exercise (for me) was spent trying to understand the fundamentals, so I didnt really progress to solving the actual problem. The backdoor exercise was informative.
The morning exercise discussion was excellent.
The mechanical and coding exercises today made it very clear what was going on and helped solidify the concepts because I was able to take time and let the concepts sink in or ask Claude to link concepts I didn't get on the first pass.
Stegonography protocol exercise
The balance across theoretical treatment, empirical results discussed at a high level, and hands-down herdr exercises worked really well for me.
A good mixture of the materials both in theory and practice. The interactive discussions between the participants were especially helpful to understand the concepts thoughout the lecture. A good engagement with the lecturer was also a plus.
Reading and exercises
I liked recreating the paper in the end.
I enjoyed the exercises, we spent time thinking about it in groups. Perhaps we could do more exercises, but timed for sure to keep everyone on track.
The presentation was mostly clear and understandable.

Would improve16

Present the definitions of concepts before using them
more claude code exercises like the one we had in the end,
Material was very interesting, but I had no experience in cryptography at all, and much of the set up and intuition seemed to rely on this. I would have appreciated a brief intro to some of these methods in isolation before applying them to the LLM. For example: I didn't initially realize that taking M XOR K was a form of encryption until someone pointed this out to me. Taking K to be a random variable was another non-trivial extension. Applying this then to the LLM was confusing, because by then I was still unclear on the fundamentals. I recognize however that this is because of my unfamiliarity with the background material. I'm sure that someone with a stronger CS background would have followed just fine.
Learning Claude vibecoding should be well-separated from learning the lecture material, or I have to do through a "boring" exercise to get to a useful one
I could have used much more guidance with the downloads. I thought I had set it up before the Intensive, but was so slow in matching the various software I feel way behind. So maybe a pointer to what, exactly, should be set up before the afternoon session.
Would appreciate much slower pace for foundational material when introducing names of variables and then speedier overview of derived concepts.
The coding exercise in the first half could have been ordered to be before the derivation or maybe both could have been given at the same time. Also a lot of people around me spent a lot of time installing Claude Code and other tools so a heads up about it would have sped up the exercises considerably.
Would be nice if we could learn more concrete examples on steganography and backdoors, what are some common protocols out there (and how they work in practice). How do "harmful responses" produced by LLMs relate to hallucinations etc.
First half the content was cool, but it was a bit hard to understand! Motivation could have been clearer. But good overall
Maybe some more structure prerequistes would have made the session better. It would have been good to have the coding tools like herdr working beforehand
The slides were a bit chaotic imho, and we jumped into the maths fast, while an intuitive concrete example would have been useful at the start :)
The coding exercises could be truncated.
I think the presentator was really fast
In a lot of the slides about Chain of Thought monitoring, a bunch of graphs were shown with no explaining text. It made it therefore difficult to follow if I missed a sentence the instructor said. Otherwise, it was great!
The explanation of the optimal steganography algorithm with a shared source of randomness was unfortunately very unclear. I spent a lot of time understanding it after it was given, and I have the impression that most attendees simply gave up. It would help if the explanation was rehearsed, and if the instructor had a dedicated place to write (or had slides with pauses prepared) to give a step-by-step detailed derivation. In the end the concept is simple to explain, e.g. with pseudocode, so it is really a loss that is was presented in such a confusing manner.
I would improve the Claude code component. I think it was a little too open-ended. Perhaps asking us to replicate a specific figure would be better. Also I think the presentation on steganography would be improved by including a very concrete example that instantiates all the variables and such.

Changed my mind9

I was already familiar with the main concepts as information theory and I enjoy to learn the many new things we went through.
Steganography can be used to secure a system, not just a threat to AI safety
I was skeptical about text watermarking working for frontier models without it degrading performance, but after today its seems possible, mathematically speaking at least.
I did not reallised essentially perfect steganography was possible with LLMs
I learned a good amount of knowledge about how watermark works in LLMs and the mathematical views of stegonography.
That i should care about encoded messages
I just did not know about steganography at all.
I really like this topic, too! I would like to know what open problems in this area are important.
No

Other thoughts2

Julian delivers content well. Thank you!
None

September 2026 Tue 29 Sep Stephan Wäldchen 23 responses energy 3.4/5

Rated against what each person expected. 0 is as expected.

Overall +0.05 as expected

Content +0.29 better than expected

Clarity -0.10 as expected

Motivation +0.43 better than expected

Exercises -0.05 as expected

Readings -0.10 as expected

Other activities +0.52 better than expected

Pace +0.24 about right

Difficulty +0.19 about right

Novelty +1.10 mostly new

Read what they wrote29

Worked well10

The speaker was clear.
The game was super fun, the reading were very relevant and the exercises very illuminating (ie. the relu exercise). Excellent format and content!!
The game -- great warmup and the way to taste the subject with your own brain. This helps immensely. The exercises -- very physically coded, so I am biased.
Game was fun.
The steg game was great :)
The game
Reading then discussing
I think the structure of the day was good and I liked Stephan's lecturing.
Information-theoretic bounds on steganography H(M) <= H(K) and Merlin-Arthur verification.
The game was quite good.

Would improve12

Present more motivation for the problem at hand; e.g. begin with a cursory overview of cryptography, and how this differs.
This was an incredibly interesting day. There seems to be a lot of empirical stuff which would benefit from more theoretical contributions (ie. a la Shafi). I would really split this into two days.
(Same as yesterday:) My main criticism of today has to do with (what I felt to be) the lack of lecture time: the lecture parts could have been longer and more formal (but still comprehensive); the little formal content was in the reading/exercises, which for me is not preferable in comparison to the average modus operandi of the previous weeks. I'm not being exhaustive about the things I liked, but I definitely have to emphasise how the way papers were discussed (all together and with slides) is a huge improvement compared to previous days.
The game -- it was fun, but lacked structure and general advice. We overthought a lot as well. Also, more work on explanations would be nice.
Similar comments to yesterday on the approach to readings. I liked today's exercises more than some previous ones. I thought the steganography was relatively straightforward, whereas the backdooring was more interesting/surprising to me. So maybe I would have liked reallocating time to go deeper into the latter.
Maybe do two rounds for the steganography game and give more time, or smaller teams? We had a bit of trouble to get organised. Should there be a pedagogical lesson from playing the game? Some truth that should be revealed from playing? Maybe say how dangerous doing research in this area is and how to mitigate the risks?
Today it was a bit exhausting. Some of the slides where heavy in formulas and it was difficult to retain the meaning of the symbols in between slides. I think the content was great, i particularly liked that it was more concrete that what we have done other days of the intensive.
number and define equations with all their quantities on the exercise sheet , better to have more time for some reading and in group discussion than more and everything rushed. I would prefer more of this day and less debate.
So many different presentations and topics, so it was difficult to follow the whole day.
I think the material is good, and I enjoyed the lecturing, but maybe the structure? It sometimes felt more difficult to grasp some concepts than I think it should have, and I found it difficult to work out why, but I think it was mostly the distribution of time per idea - too much for some ideas and too little for others.
Hands-on implementation of undetectable backdoor constructions.
Everyone's a bit tired so I think this day could've been a bit lighter on lectures and more discussion, reading focused.

Changed my mind5

n/a
Yes!! I definitely want to work more on this kind of workk.
It made me more concerned about steganography.
discovered some labs that take students without a master, that work on safety
Paraphrasing defenses are far more fragile against steganography than assumed.

Other thoughts2

n/a
Fascinating module, really sharp math.

August 2026 Thu 27 Aug Stephan Wäldchen 24 responses energy 3.5/5

Rated against what each person expected. 0 is as expected.

Overall +0.58 better than expected

Content +0.50 better than expected

Clarity +0.42 better than expected

Motivation +0.79 better than expected

Exercises -0.05 as expected

Readings +0.26 better than expected

Other activities +0.44 better than expected

Pace +0.25 too fast

Difficulty +0.17 about right

Novelty +1.08 mostly new

Read what they wrote17

Worked well7

Stephan is a great teacher!
Both the choice of content & the two talks!
group exercises worked
group exercise, interesting readings
The game was so cool! The topics presented today were interesting and I will look into them more.
The lecture was amazing: Fun game as an introduction to the topic, super clear and structured explanations. Very well chosen content, level and depth.
I really like the table schedule for the whole day!

Would improve7

Again, do good lecture notes and people study them in the day in intensive, with supervision, and that's it. Alternatively if this is too extreme, just don't make the day feel like a bluedot course meeting, like funny game then not a lot of time to actual reading on my own than empirical paper reading for 5min and discussing one idea that honestly we just take at face value because a paper is not readable in 5min, etc. This day was less confusing than yesterday but still kind of terrible, the ideas are for sure important and I even got more security/formal verification/complexity theory pilled with all of these, but the implementation of the day is not great. Louis talk was amazing, not because the talk was particularly interesting but this idea of slowly building up ideas to solve huge problems and figuring out which problems are even worth trying to solve beforehand is great. I generally think his mental model about stuff is correct and agree
Maybe more exercises (even take home) that cover the content from the first part of the day?
not enough time for reading papers, doing exercises
Great material and variation, just a bit too fast paced
I feel like today we were covering too much ground and I was getting lost very often. The day felt very dense.
The lecture was amazing, but the talk was super confusing to me: Please provide a big picture overview first and then either go into detail or leave topics out. Leaving lots of open ends while at the same time not mentioning which topics are central makes it impossible to follow. The definition of f(M') was too vague (missing domain and codomain) and could only be reverse engineered via its relation to C(M').
Exercises: I feel like the exercise text can be improved, I felt like the "relevance" was a bit hard to see. Also solution d) is missing (not like there's a lot one needs there) Readings: Not enough time. I think it would be useful if Stefan was a bit more specific on what we should read given the time constraints

Changed my mind3

I'm more ARC theory research pilled (their new stuff)
feasibility of planting backdoors in LLMs
understand how steganography is applied better

June 2026 Wed 1 Jul Stephan Wäldchen 11 responses energy 3.3/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.5 of 5

Teaching and format 4.3 of 5

Difficulty match 3.0 of 5

Energy 3.3 of 5

Read what they wrote10

In their own words9

I think that the game was fun but maybe too long. I would have preferred to spend more time with the theory
- The game in the morning was enjoyable and illustrated the difficulties of coming up with a steganographic code - The backdoors topic was fascinating - I liked the dynamic of reading papers in groups and then coming together to discuss the high level ideas of the paper. It allowed us to digest many more take-aways in less time - I missed having a short break in the morning - The break in the afternoon at 14:45 was unnecessarily early in my opinion. Instead of doing two breaks in the afternoon, you can do just one but in the middle of the afternoon (e.g. 16:00-16:15)
I found the morning exercise of steganography fun, but taking up a bit too much time maybe. The lectures and discussions were good.
The game was a nice intro for the day! The limits of interpretability talk was excellent but a little depressing. I got a lot out of the talks earlier in the day too. It seemed like we were in a bit of a time crunch (perhaps due to our tendency to draw out breaks (: ), so I think cutting one of the reading sections for more problem set time would be a popular decision.
Stephan made the first half of the day quite enjoyable, just like yesterday. Louis' lecture was also good but maybe the motivation for the compact proofs research wasn't that clear. A good day again in terms of the mix of lectures, discussions, exercises and the game in the morning.
steg game was fun! pretty much all the afternoon content was pretty meaningful. I would not change much today, except, maybe somehow make it a bit more interactive, although idk how this would be done specifically. I liked having the quick problem se session, and if another could somehow be included, I think that would be nice.
Good use of table, good explanations, overall time and energy management was quite good. The talk about explainability was nice in that it connected to the methods we did in debate and steganography, but in itself the topic may be not as relevant for AI safety.
Good session, nice topics, exercises were instructive. I found the weights extraction via steganography very interesting.
I found the initial lecture somewhat boring. The activity (which was designed to be fun), the readings, and the discussion prompts today were well designed and fun. The second half (after lunch) was very good (peak) - the lectures and the exercises were great. I would like to focus on the exercise sheet, which I believe was very well designed and had a very good learning curve. Today was pretty productive.

Other thoughts1

last two days(june 30, july 1) were well designed.

April 2026 Sun 26 Apr Stephan Wäldchen 13 responses energy 3.4/5

Called “Backdoors and Steganography” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.2 of 5

Teaching and format 4.6 of 5

Difficulty match 2.8 of 5

Energy 3.4 of 5

Read what they wrote30

Worked well11

I thought it was very useful to work through the ReLU exercise and present solutions on the board.
Discussion of papers with TA's was useful to make sure we got the important ideas
Starting out with a game was a great way to be introduced to the core topic and be engaged. The rest of the day being more chill with readings and lectures was enjoyable.
The lecturers as a pair were great they were extremely entertaining. Stefan’s(?) concluding talk was great.
The steganography game was excellent.
The cohort was jolly. Stephan was ebullient. Louis was perhaps more relaxed than when he was flying solo.
The game in the morning was fun. A little silly, but got us to think about simple approaches to steganography that LLM’s are also using - it combined nicely with the reading in the morning.
Very engaging activities and teachers/content. Readings and exercises were also informative and at the right level of difficulty.
The game was fun, and also positioned at the right point in the schedule (i.e. the start of the day).
Initial steganograph game, range of activities/exercises
Interesting topics and papers, and I enjoyed the varied teaching formats of the day.

What they took away9

The backdoor paper was very interesting to me.
Backdoors can have serjous security consequences, but implementing them requires white box access to weights and relies on minimal downstream SFT or complex large scale data poisoning for frontier models
That chain-of-thought in LLMs can quickly become unreliable and is not a way to actually see what the model is thinking, and that it can be used to pass secret messages to future versions of the model.
I think I gained some important mathematical intuition during the contrived backdoor single hidden layer example that is an important thing taken forward.
Being elicitable =/= being detectable. Superposition makes it harder to exploit and discover structure.
I finally learned why we’re told not to put optimization pressure on COT. Also the Merlin/Morgana setup in the afternoon was interesting for interpretability.
I was completely unfamiliar with the concept of steganography, so everything was useful. Thinking through how this can be used for nefarious purposes, and how this is fine in practice was a particularly useful exercise
Various nice ideas for this, with the steganography intuitions, backdoor intuitions, and Merlin-Arthur approach
Steganography for weight exfiltration. NP-hard backdoors.

Would improve10

Perhaps would be good to talk more about the assumptions that need to hold in order to make backdoors impossible ro efficiently discover and how much this maps on to reality.
Encryption game was entertaining but idk if it was particularly useful, also exercises seemed a bit contrived and uninformative, but interesting, maybe could have tried to implement or test backdoor techniques in toy models representative of the readings?
More whiteboard solutions of the exercises on would be nice.
Steganography game was a bit too easy I think a really hard version where the team had to agree on their secret key before they know, not just what the word and question is, but what the medium is like it could be revealed that they have to communicate through some constricted alphabet with some constricted grammar (this teaches something about Schelling points that I think was missing today a little). The other hard version is the obvious one which combines the ones today to write arbitrary messages after paraphrasing.
Finish on time.
Stephan and Louis both expressed a wish to spend more time improving the exercises they had written quickly, for yesterday and today. The long steganography game at the start of the day may have had non-didactic value for bonding or boosting energy, as a dance break would do, but it certainly wasn’t necessary to get the idea across.
The afternoon exercise was a bit too computational… maybe there’s a way to rephrase it in order to emphasize the structure/intuition more?
I think I find the 1) reading paper and 2) discuss not the best way to get across the materials, also hard to get an idea of what would be treated today as we did not have a master doc etc.
The last presentation was interesting, but it's link with the current state of the field could have perhaps been made more explicit.
The mathematical exercise can probably be made more relevant to the backdoor ideas - I don't think it really conveyed that much too me beyond the intuition of relatively small changes allowing fairly high probability of separation (although I still enjoyed just doing gentle maths for a bit :))

Training Dynamics 5 runs 133 responses, 169 written

for pasting into Claude

October 2026 Fri 16 Oct Charles Renshaw-Whitman 35 responses energy 3.9/5

Rated against what each person expected. 0 is as expected.

Overall +0.91 better than expected

Content +0.91 better than expected

Clarity +0.88 better than expected

Motivation +0.82 better than expected

Exercises +0.29 better than expected

Readings +0.25 better than expected

Other activities +0.61 better than expected

Pace +0.09 about right

Difficulty +0.18 about right

Novelty +0.47 mostly new

Read what they wrote40

Worked well13

the lecture by Daniel Kunin was totally amazing: clear and inspiring and presenting a compelling argument for why training dynamics matters for safety. the part on the whiteboard was especially great!! and i wish there had been more of this during the intensive. it forces the speaker to progress at “the speed of thought” and we can all follow along
The second lecturer
The morning exercises were a great preparation for the afternoon's lecture.
The level of mathematical formalism was was tuned just right (for my background/taste) so it it did not obfuscate the larger point.
The first morning session could have been more lecture structured. Otherwise, the second half was great.
Guest lecture by Daniel was fantastic. Very clear and interesting.
Daniels lecture was one of the best talks ive heard
Discussion and drawing on whiteboards.
Both Brianna and Daniel were excellent in their delivery of the information. The exercises, as well, we excellent.
I thought working at the board was good.
The lectures were well prepared.
the white board discussion and motivation via toy models was perfect, I would encore more speakers to attempt this format, it lands well with mathematicians and physicists
Nice to start off with the basics. Doesn’t hurt to repeat some stuff from time to time.

Would improve17

It was great today!
Teaching was excellent (in the beginning it was very high level, but a good overview)
I found a lot of this stuff hard to motivate from a safety perspective. I think some more motivations would help me stay focused.
it would have been helpful to know that the morning lecture was presenting things that were going to be used in the afternoon lecture. Instead, I got the impression that the content of the morning lecture was irrelevant in a very broad sense. I remain a puzzled by Charles’ and Brianna’s approach to teaching in the intensive. the votes below are strongly influenced by Daniel’s amazing afternoon lecture
Very engaging. The split between lecture time and exercise were ideal.
Maybe more talks today would be helpful! There was a chunk of time where I wasn’t quite sure what I should dedicate to.
First lecture with more info and more precise
nothing much, the white board presentation was extremely nice
For today, a bit more of the big picture and big results upfront to motivate what is going on later and guide exercises. I think also the afternoon's derivations could have been sped up had it been made explicitly clear during the morning that exercises were crucial.
Not much
Almost nothing to improved. Perhaps some embedding GIFs displaying function space changing through training epochs
the content was kinda light in the morning, and kinda compressed in the afternoon. that could have been distributed more evenly. also, some guidance or coordination for doing the exercises could be helpful, so there are groups to collaborate with.
I think one of the lectures went quite long and a break could have been good, but it seems like people were excited and interested.
Both lectures today were amazing.
The second half was a bit dense and didn't have any breaks. There was about an hour left afterwards so the time could have been better utilized.
Not much, Daniel was a great lecturer.
Tiers for difficulty (easy/hard)

Changed my mind10

My main takeaway from Daniel’s talk was that initialization matters, I’m still trying to parse through exactly how initialization affects the output in transformer based networks as opposed to the linear networks he gave the talk on.
I feel more that there's a bag of 5 observations that a lot of people are coming up with increasingly divergent theories to explain. I genuinely enjoyed learning the material today but it makes me think there are higher priority areas for research when it comes to AI safety
Mysteries of Deep Learning and Training Dynamics have emerged as the central scientific interests for me.
Learning mechanics seems like a very promising theory for Alignment from first principles.
Lazy vs rich learning. Studying these with higher dimensional parameter spaces.
I finally understand what's up with Lazy / Rich regimes.
I wasn't too interested in linear networks before today but I think I will look into them a bit more.
Daniel's lecture on rich learning completely changed my mind about how generalization happens during optimization. This was a very fun lecture to watch.
Daniel mentioned that the black box nature of LLMs comes from the data and not the algorithms, and that makes a lot of sense in hindsights.
It was mostly new.

September 2026 Wed 16 Sep Guillaume Corlouer 37 responses energy 3.6/5

Rated against what each person expected. 0 is as expected.

Overall +0.09 as expected

Content +0.30 better than expected

Clarity -0.06 as expected

Motivation +0.18 as expected

Exercises +0.12 as expected

Readings +0.06 as expected

Other activities -0.04 as expected

Pace 0.00 about right

Difficulty 0.00 about right

Novelty +0.91 mostly new

Read what they wrote47

Worked well14

The exercises were good
Really enjoyed the content on learning dynamics today, was mostly a review for me but was well taught and found the exercises a very useful review.
paper reading + discussion with people after is always sick, we should have this every day imho
working out the examples and discussing the paper with the other intensivists :)
Deep linear networks intuition for lazy vs rich regimes.
The day being split into a lecture half and an exercises half was really nice.
The paper activity was great. I like when we have exposure to papers and read them properly. I also like how some days continue into others, like today was a great supplement for Physics of Deep Learning on Tuesday, and a nice continuation at that. Also, liked mentioning Noether Theorem and the approach to the problem via symmetries -- very physics-friendly. Allows to look at the problem from different angles and using different approaches, which is mathematically invaluable.
I liked that we had a good mix of theory, exercises and empirical analysis of the applications. I think this day had the best split between these three of any so far.
The topic is very interesting. But the materials provided to absorb the content and learn was not. It would be good to rethink this!
I liked the edge of sharpness/learning rate limit result.
exercises reading person seating on the LHS who explained some things more clearly than the lecturer
The reading lists were useful, but I believe it was overly ambitious to reach the stage of informed group discussion after devoting so little time to actually reading a paper.
Lecturer very clear.
Reading by myself.

Would improve21

Exercises were surprisingly easy. Maybe remove hints? I think splitting the lecture in two would be better for maintaining the overall attention of everyone. Some parts of the lecture were suddenly very mathematical and some of the important notions weren't reviewed. It would be good to give a bit more context on Lipschitz stuff?
More time on NTK derivation steps.
Split: theory morning - exercise afternoon is not optimal in my opinion, I did enjoy the exercises and coding a linear network was also useful to see the theory in practice.
Have more that one speaker on a given day. I think I'd prefer having two/three topics per day with each topic's content spread over two/three days.
I think two lectures in the morning one after another was quite intense. The material was good and the delivery too, but it needs slower pacing to sink in. As I caught myself thinking on one of the last slides "wow that's a lot, can we please stop here," as each slide needed some time to be digested. Yesterday was an exemplary structure: lecture, then exercises, then lunch, then lecture and exercises again. Today would've benefitted from it too. I would make exercises slightly, just slightly harder. Perhaps there was no need to make the matrices diagonal, or maybe there was. Also, on paper activity -- maybe having a small workshop (30-90 mins) on paper / LW post reading (first (and last) sentence of each paragraph, then in full) would be beneficial. How to read with the best retention / understanding given time constraint and objective.
I found the lectures were quite difficult to follow due to the density of technical material, but it did nevertheless help clarify the reading later in the day. I don't think the content itself was necessarily too difficult, but I struggled to grasp it from the lectures at least. Perhaps due to too many interruptive questions from the audience. With the reappearance of the NTK, I think in hindsight it would have helped for it to be a larger focus during the Physics of Deep Learning day. I discussed it a little with others, and I didn't feel convinced any of us had understood it deeply.
- the visual (slides) and verbal (speech) parts of the talk were a bit too disparate, which made it difficult to follow - it would be nice to have a subset of exercises to prioritise (on top of this, the fact that before starting the exercises I wanted to review and understand the slides took me some time) - like yesterday, I feel like last week's average day structure was preferable to the way today's structure was implemented (in particular, I prefer more talk time intercalated with other activities) PS: I think that less time for exercises makes the "lecturer" think them more carefully, rather than assigning an unattainable amount of exercises
I would split the lectures in the afternoon and morning and maybe talk a bit more about the stochastic part
I'd like to cover more material during the day. Going over differentiation and notions of gradient and hessian over the day on a course for math and physics PhD graduates seems questionable -- it's like a one undergrad university lecture in a day. Going over some more experimental exercises could be great as well
Using Claude to generate slides is fine, but these have to be checked. Figures are only sometimes cited (in vary different forms), making it difficult to go to the original figures... Some figures I couldn't even find after looking at all the references in the references slide! Also, the exercises were "fun" but they weren't super useful I think. It was valuable that they showed us the maths of the results we saw during the lecture. However, there was too much focus on this which meant that we didn't have enough time to read and understand the papers. Change this!
I think it was a bit optimistic to read some of the papers in the given time. Maybe a guideline to each paper would have been helpful.
I thought the day was quite good. Maybe add a more practical exercise.
Maybe slightly too much discussion during the lecture. I would have preferred to cover the essential ideas first before going into "what if's". As it was, I lost track of the bigger picture and the current train of thought within the lecture material multiple times. The fact that I was tired certainly didn't help. I also think that it would have been great to have this sooner after the "physics" day, since many of the ideas from that day were picked up here.
General feedback: Some people today kept asking questions that were tangential, persistent or not particularly good and this disrupted the flow of the lecture and made the experience overall worse for most people in my opinion. Given the material covered each day is a lot, I think it's important to stick closer to the schedule and lecturers can answer such questions after the lecture; maybe the participants should be explicitly encouraged/reminded to keep some questions for after the lecture?
maybe adding a small project for people to train and compare their own deep linear networks.
there was not enough time to read the paper - the discussions were not very productive because everyone was just lost and confused
I'd like to see some more fine-gradined look at some papers, examples of training dynamics and maybe a few notebooks to reproduce experiments and play with toy tasks without reproducing them with Claude.
Unfortunately, the main lecture venue is impacting my ability to enjoy the taught sessions. The screen is small, we are crammed with no desk space, and it is hard to hear the lecturer. Any improvement on that would be greatly appreciated. Regarding the lectures, Guillaume did a good job at structuring the material, It took me a while to get the hang of notational conventions, but it was fine in the end.
I thought today was very good. I would have liked a lecture in the afternoon.
Too long lecture sessions without a break. I was not able to follow. Also the time for the reading was too short, we ended up discussing only section 1 of the paper.
The talk included a neat survey of a bunch of results that are out there in the literature, plus a more detailed discussion of some properties of deep linear networks. The exercises + reading gave a pretty complete and concrete understanding of the basic results about the latter. It was a little hard for me to understand the significance of the several papers that were mentioned or guess which I might want to read. I suppose I'd also like to have a little understanding of what the big open questions/short-term goals of training dynamics research look like.

Changed my mind9

Grokking understood as phase transition rather than mystery.
I'm more convinced today about training dynamics's potential as a helpful tool for AI safety. That said, I'm still a bit unconvinced that the mathematical frameworks introduced thus far are helpful for alignment. I'm yet to see how they can interface with empirical work in a way that allows theory to be developed in a way that compliments experiment.
The usual -- Learning Dynamics might be lost in math at times. As a theoretical physicist, I appreciate the bottom-up approach of going from simplest to harders, but as a theoretical physicist I know that it tends to lead into focusing on the simplest too much due to math being tractable.
I think what we covered today was very interesting, and I can see how it could help with understanding neural networks.
I'm interested on what was mentioned about tensor networks. I will definetly take a look into it.
Misalignment could be a more general phenomenon than I expected previously.
Given my previous background in optimization theory, I struggled to find the most promising application of it to AI safety research. Training dynamics seems to be the most promising right now!
n/a
I learned about Swiss AI safety days, maybe worth applying.

Other thoughts3

We need a projector for lectures
Great module.
n/a

August 2026 Tue 18 Aug Guillaume Corlouer 34 responses energy 3.6/5

Rated against what each person expected. 0 is as expected.

Overall +0.68 better than expected

Content +0.82 better than expected

Clarity +0.44 better than expected

Motivation +0.32 better than expected

Exercises +0.30 better than expected

Readings +0.21 as expected

Pace +0.12 about right

Difficulty +0.12 about right

Novelty +1.03 mostly new

Read what they wrote39

Worked well11

good readings and exercises
Everything else. In particular, the slides, exercises, and readings synergized, more so than on other days so far.
Lectures were the best so far. The readings were interesting (but hard).
Exercises
Just the topic was cool.
Exercises are good and I like hands-on math. Good mixture of exercises and readings
How structured everything was. Having the material in advance, even just 12 hours before the lecture makes a huge difference
The material was very accessible and well-explained by the instructor and it was helpful to chat to the TA.
The discussion on how depth initialization affects the dynamics
The reading where we actually had to write up our thoughts was really effective at getting us to focus the discussion.
The mathematical exercises, though pretty easy to solve, were very nice and gave me intuition about what kind of behaviour DLN training is like.

Would improve14

a lot of the math is dense if newer to this field
The Saxe et al paper is kind of bad, didn't thought it was worth it. The exercises are super great though, I could only do 3/5 because I spent some time talking to Adam from Simplex, so maybe giving more time to finishing the exercises instead of reading this paper would be great (or something else that is not this paper). In general, I think this training dynamics day was one of the most informatives (I didn't have previous context on it)
I think the morning slides could've made it more explicit that i) the training algorithm converges to the true solution for this problem, ii) proving this and bounding the rate of convergence here isn't our focus, and instead iii) we care about _how_ it converges for reasons related to alignment, and this will be the focus of most of the slides.
I think the exercises could be improved - I think having more text in the statements motivating/explaining the NTK and the connection to saddle points would be nice (such as the explanations in the solution guide).
Morning was a bit lecture heavy. Adding exercises in between could slow down the pace a bit.
I felt like the lecturer sometimes didnt quite answer the questions.
I don't know if the Adam/Paul thing today was planned beforehand, but I would've liked if this had been announced a bit earlier and I think one could have worked around it a bit. I was a bit unsure today if I should focus on talking to Adam or do the exercises (I can only talk to Adam today, but I can always do the exercises. Though *will* I?)
I would have loved some more challenging exercises. A lot of the exercises were just one line solutions. They didnt stimulate my brain much.
I think the slides have been pretty bad. The slides are just bullet points, but the properties of a dynamical system seem to be visual, so would be nice to have some more focus on intuition.
I enjoyed the AMA with the Simplex thing, but it felt hard to return to exercises rather than think more about their research agenda.
Make it more structured and clearer channels of communication. Cantor Hall was abandoned from lunch and I didn't really know what I was supposed to be doing.
The reading list was well-curated, and I appreciate the teacher posting the prerequisite announcement early with clear communication. The only issue was that the preparation window was quite short. Fortunately, I had already read some of the Saxe lab's work around February for their interview, which allowed me to follow some of the lecture. I like the reading group exercise in the afternoon as well.
Today was really good, I don’t really have any critiques.
Can't really say much as I was not feeling too well. The one thing that was not so clear I guess is that we were using a very simple toy model (DLNs) while actual frontier models do not use this architecture. I would've appreciated some more stuff on TTNs as the better toy model to study, being both non-trivially expressive and mathematically tractable to investigate.

Changed my mind11

This seems an honest shot at understanding NNs. I'd be very up for studying similarly toyish models but increasingly complex with these techniques. More neglected than SLT and comp mech for sure (although comp mech still seems more promising)
55% grad school 45% ai safety. I feel less confident that an understanding of deep learning helps us solve alignment though. I read an argument yesterday that a complete understanding of mech interp would help AGIs improve themself.
The math behind dln is both simple and powerful. I feel natural extensions to nonlinear networks should be very fruitful for agi safety.
I was not expecting this to be interesting (thought it would be more engineering) but was suprised at how interesting DLNs are. cool!
I am really surprised how much you can milk linear models.
Today felt like a necessary step of bridging theory and practice in my mind. Theres still work to be done there but today closed the gap a bit, especially after reading the geometry of fibers paepr
This seems to be slightly more relevant to safety than I originally thought.
I had some good discussion with Zach and found it was helpful to get a sense of what people were thinking about
About how difficult it is to move from deep linear NNs to deep non-linear NNs.
The clarification that SLT models a static landscape was crucial for understanding how these geometric concepts connect to training dynamics.
I had never paid much attention to training dynamics because I had always seen it as something quite opaque, whereas now I see it can actually be analysed quite precisely and in a way that feels much closer to the real world contexts than some of the other things we have learnt about.

Other thoughts3

I would really appreciate if someone told people to come to them after the lecture when they ask their third follow-up question that is clearly irrelevant to better understanding what the lecturer is trying to convey
N/A
I wasn't feeling particularly well and couldn't really make the most of it but I still liked the way it went! I left home early after solving the exercises.

June 2026 Thu 11 Jun Guillaume Corlouer 14 responses energy 3.1/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.1 of 5

Teaching and format 3.9 of 5

Difficulty match 3.0 of 5

Energy 3.1 of 5

Read what they wrote10

In their own words9

Good presentation and interesting topic Exercises were mostly interesting but some too much plug in and simplify. Time to read paper was too short, so discussion groups did not really form. Maybe from the beginning organize into groups of people who want to read either the same or different papers to then discuss.
Excellent choice of material and clear exposition, well-calibrated exercises and interesting papers to read as a follow-up. The time to read the papers and discuss was too short - a lot of people just ended up reading for the full hour +. But I don't have a problem with that. Guillaume has done a nice job!
Interesting topic. The content was good but the exercises were even better. I liked that today we got more time than yesterday to really sit down and do problems, and in the process flesh out our understanding. The paper reading part of the day could've been a little more organized but people's energy also starts dwindling by the afternoon (especially given it's thursday!) so the open-endedness fit well. In terms of improvements, before the lecture in the morning, some background reading on the Neural Tangent Kernel could've been helpful.
there was too much choice in readings and exercises so it was difficult to discuss later on. a little bit more structure could be helpful, less choice or to agree beforehand that we'll present to each other what we've read once we meet
I really liked the exercises, they actually helped me understand the material as I spent time with it. I did not understand much in the lecture except what I had already read. The exercises and the results in them were the most interesting today (only did until question 3, need to do the later ones). On my side, I would improve the preparation I make before the session starts, and on the program's side, I hope some more time is spent with the material. Feels like too much happening in a few hours. Maybe that is the purpose, not sure.
- I loved the reading. The papers were very interesting - I feel like we didn't have enough time to read the papers properly. I would have preferred more focus on reading and less on exercises.
Having a choice between papers to read was nice! The lecture was great too. I really think, again, that more group work would have made the exercises more productive.
Today's content was very informative. The presentation given in the morning session exposed the main ideas and concepts of deep linear networks. The exercises were also helpful and doable. It might be useful to add a hands-on session in the schedule, so that participants can see how the concepts exposed in the talk can be obtained in practice. I'd have appreciated if the tutor was more engaged in the afternoon session, maybe giving inputs to steer the discussion (similar to the SLT module that took place yesterday). Other than that, the suggested material was very interesting and in line with the content of the module.
interesting new perspective, the teaching was less structured than usual, was fine, but maybe more structure would be better

Other thoughts1

Meta - feedback: maybe instead / in addition to easy/hard you could measure how well-aligned the material was to the participants' background. eg. today I wasn't bored reading the "GEOMETRY OF THE FIBERS" paper, it was a comfortable level of challenge, but it was very aligned with my background, so "easy" in the sense it was familiar. By contrast, some of the reading on the first day was challenging because it was quite far from my background, but the concepts themselves weren't too hard. Maybe it's a distinction without a difference

April 2026 Sun 12 Apr Guillaume Corlouer 13 responses energy 3.6/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.5 of 5

Teaching and format 4.0 of 5

Difficulty match 3.3 of 5

Energy 3.6 of 5

Read what they wrote33

Worked well12

I liked the freedom to choice between readings based on interest (at the end of the day).
I am in awe of all of these humble, knowledgeable, enthusiastic teachers, specialised in each topic. How did you find them? What are you paying them? There must be some dark magic afoot. This bit may sound silly, but I also felt much more comfortable with a lower body density in the room.
Teaching format again worked well, lectures conveyed the meaning and intuitions fairly well - thank you!
Nice exercices
Lectures were really good
The presentations were great and the exercises useful
The exercises to illustrate regimes in training dynamics were great.
We had more time to go deeper into the readings which really helped to learn a lot about the topic.
Lectures were very good and the format was also fine
Studying by myself at my own pace. I managed to go a lot deeper than we managed to in the lecture/in person session, even though I spend much less time, since I was able to basically skip what felt very clear to me based on my background and focus on the problems where I just completely/partially lacked the background.
Exercises were representative of the work to put the paper, and presentation
Liked the presentation today, the format was digestible. I did find some parts hard but it was nonetheless helpful

What they took away12

That grokking might be related to a change between rich and lazy training regimes.
The difference between strict and non-strict saddle points
Understanding the lazy vs rich perspective better, where this was reinforced through the first lecture, exercises and final paper reading
Gradient flow and implicit regularization
Usefulness of SVD in modelling
It was extremely useful to have a non-SLT perspective about the implicit regularisation story, particularly having a dynamical one.
What parameters can control different regimes in training dynamics and grokking.
Grokking, that was quite surprising and interesting to debate with others.
All of the Saxe dynamics exercises were incredibly useful. Very good choices of topics today and the lecturer was very clear.
It's exponentially more likely that you'll have saddle points in higher dimensional NN spaces and that predicts that you are almost always just in a "slow/saddle" phase and it's only a matter of time until you move to a different position if your feature space is very highly dimensional.
analytically tractable models are hard
Understanding divergence, and how/why some training dynamics can alter the model

Would improve7

Sometimes questions from the audience lead to discussions that lacked clear focus and relation with the topic. Maybe these questions could have been prevented by starting with a more precise demarcation of what we can and cannot hope to learn from studying training dynamics (in the short term), so that people know what to expect.
I think the lectures could have been a bit more concise, both on the side of what was said, as well as less participant discussion. (As a very minor point, Guillaume could speak a little more quietly in a relatively small room like this)
I would benefit from more elaboration on how this picture is distinct from SLT and ways in which it focuses on a more general set of possible explanations
I would have liked to go over the math exercises in a more rigorous way where we show the solutions on the whiteboard for example.
Give people spaces to work quietly at their own pace and implement reminders for them so they can try out whether the next session is useful for them bcs it assumes the amount of context/background they have. This also adds time limits, which makes it much less likely for independent students to procrastinate.
shorter blocks of exercises/reading between check ins
I personally like a quick review of the basics, but given everyone is very advanced, I don't mind there is none

Other thoughts2

Thank you! (I've realised my feedback forms are likely not expressing the gratitude I feel where I'm writing them more plainly)
I love Iliad ♥️

World Models 4 runs 112 responses, 167 written

for pasting into Claude

October 2026 Tue 20 Oct Alec Boyd 34 responses energy 3.7/5

Called “World Models and Transducers” that cohort.

Rated against what each person expected. 0 is as expected.

Overall +0.33 better than expected

Content +0.27 better than expected

Clarity +0.55 better than expected

Motivation +0.21 as expected

Exercises +0.12 as expected

Readings 0.00 as expected

Other activities +0.25 better than expected

Pace -0.09 about right

Difficulty -0.06 about right

Novelty +0.45 mostly new

Read what they wrote39

Worked well14

frontal class with a lot of partecipation and questions from the audience plus nice exercises
Questions to the audience
The birds eye view in the end and applications to multi agent systems seemed like a good perspective on how such theories could be practically relevant for safety, even when we do not have a full theory
Possibly the only day so far where we were not rushing to go through content, and definitely the only day I was able to complete all the exercises in the alloted time.
the exercises went well for me, but I was not a fan of the material in general.
The exercises were very well-chosen to reinforce the theoretical foundations covered in the lectures.
Working together in subgroups on particular exercises. Enough breaks (and suggestion to go outside for a while). The presentation itself was just well-paced and I liked the simple examples to really get the hang of the processes ourselves.
great pedagogical approach, putting things into context, building the basics, and then eventually getting to complex applications
The lecturer was very good: kind, clear, engaging.
We discuss about a possible project together with the mentor. I am very happy about it and I look forward continuing working on this!
Some content reviewed Mondays lecture but I liked it, it gave time to properly digest the material, and also having 2 different people explain similar concepts gave nice perspective.
You might find this in many of my feedbacks, but I enjoy doing the exercises! During the time allocated to exercises, Alec was pacing the room and offering to help out (not that this is necessary, all mentors have made themselves very available) which made it easy to ask the simplest questions!
It was a good idea to introduce transcoders more formally. We had seen something like this the day before in Paul's lecture (where he constructed belief states using a Bayesian update rule), but this helped to make this concrete.
I liked the presentation style and motivation. The presenter was very clear and deliberate

Would improve16

nothing it was great
The lecture in general had a lot of overlap with the one from yesterday and hence at times was a bit boring/too slow, especially in the beginning However, the later exercises was a good reinforcement of the things we did yesterday
The concepts today felt more intuitive than that of Day 11 (Computational Mechanics), and would have helped to see it before Day 11.
This day had a remarkably large overlap with yesterday's topics. In some cases, it presented the details we wish we had the day before; in other cases, it felt like an unnecessary review. It is nice to spend two days on related topics, but they should be better coordinated together.
I was not a fan of the topic, so I quickly did not follow and got lost. Perhaps this topic is a little out of place and should be skipped?
It would have been nice to focus more on the implications when applying this analysis at scale for LLMs and factors/multi-agent systems
I only attended up to Section II - up to that point this was interesting but the upshot hit a bit less well as I feel like world model theory is already implicit in the computational mechanics lecture, so to a certain extent it *felt* redundant.
This felt too overlapped with the computational mechanics day. A lot of things (including the MSP exercises) were extremely similar to yesterday.
The lectures are nice but would like to see how the things we learned can be applied to understanding LeWM and other SOTA world models, not just toy examples
The lecture was well structured. Went from good overview to very new studies. One improvement would be to include more codes to practice. There was one exercise with coding. I would increase the proportion.
I would make the content of today more independent from that of yesterday.
this was great!
At times I felt we got a little too in the weeds and I was struggling to understand how this relates to the big picture of alignment and what we are trying to figure out to make AI go well. That being said, he was a great lecturer. The exercises were helpful, he asked probing questions, and he was very nice and easy to talk to.
Maybe some more coordination with the previous day to prevent overlap
This lecture was really useful, but it seemed like there was a lot of overlap with Paul's lecture from the day before. It was really helpful for me to see this again, and I think Alec made a really good choice by introducing the topic in the language of transcoders. I'm really interested to understand what this formalism has to say about multi-agent interactions, and I'm looking forward to hearing more later on.
I think this lesson was redundant from yesterday

Changed my mind8

I feel that studying multi-agent systems is a bit more feasible and am a bit more optimistic about practicality of breaking down the enormous complexity in AI systems into interpretable parts.
By this point I've already developed an interest in SLT/mech interp/computational mechanics, but am starting to get a bit overwhelmed by the material and want to focus on how I can realistically best make an impact in this field versus broadening my horizons even further then going back to my status quo. Given my more technical background vs other participants, I feel like my path to impact in AI safety would initially be in the research engineering route while keeping up to date with AI alignment research progress vs driving it theoretically. I *am* considering a MSc and research supervised by academics in SLT (already in discussion) but I feel like short-term I should aim to stay in SF and continue engaging with Mox/Iliad. As such, I've been applying to jobs (in data/evals/post-training) in SF and am currently in a couple interview processes, and hoping to informally re-attend future Iliad Intensive lectures and cover gaps in my knowledge in a longer time horizon if that's okay with the organizers.
Automata can actually be an interesting approach to understanding agents.
cool research! i was unaware of the material from yesterday and today and how interpretable this aspect is!
The multiple sub-agent picture seems like an interesting line of work for understanding AI.
Comp. mech seems very interesting, might be what I work on if I go for AI safety.
I was surprised to learn that the Geometry of belief states reflects geometry of prediction, and it had me thinking about Paul’s talk on belief state geometry the previous day. I haven’t had enough time to mull over this to create a project idea out of it yet, but I certainly have questions I’m thinking about!
I came away pretty confident that "transformers encode belief states in the final layer of their residual stream" is a solid starting point for understanding transformers.

Other thoughts1

just as a suggestion: there could be a day on autoformalisation -> ai-maths for ai-safety

September 2026 Fri 11 Sep Fernando Rosas 39 responses energy 3.7/5

Rated against what each person expected. 0 is as expected.

Overall +0.30 better than expected

Content +0.05 as expected

Clarity 0.00 as expected

Motivation +0.05 as expected

Exercises +0.08 as expected

Readings +0.25 better than expected

Other activities +0.32 better than expected

Pace -0.19 about right

Difficulty -0.14 about right

Novelty +1.14 mostly new

Read what they wrote54

Worked well14

i enjoyed the group exercises
The last bit about cognition in cognition in LLMs. I hope we will have more focus on similar experiment driven research.
The fact that it built on top of yesterday's material was crucial to its value. But it was a very technical day that required more time to grasp.
The exercises let me understand exactly what was happening. The slides were gradually presenting information, making the lecture much easier to follow.
- The interdisciplinary angle i.e bringing in stat phys / neuroscience (a lot more intuitive for me_ - The Colab exercises!
Fernando did a great job at explaining very technical concepts and also in giving intuitions when faced with questions.
I think the slides for the lectures were well-designed.
I thought the introductory lecture was the best introductory lecture we have had so far. It motivated the framework in a clear way without assume too much background.
- The introductory lecture at the start of the day. Spending one hour to sent context before doing anything technical was great - Margot's talk at the end of the day is a great addition. Having someone talk about what they've worked on and the sort of work being directly applied to AI safety was nice.
Presenters were clear and concise
Mixing pen-and-paper exercises in steadily increasing difficulty with coding exercises.
It built upon the notions that we saw the day before!
The last part on metacognition seemed quite interesting (maybe not too mathematically though).
The real-world examples by Fernando, such as "Listening to a conversation from behind the wall" -- precisely what I asked for yesterday. They were helping putting the discussed material in context. Also the Colab exercise helped understand the topic and rebuild the math behind all this, very good idea. Also, the lecture by Margot was very satisfying as the day closer, where we went back to the high-level thinking.

Would improve26

for the group exercises we did some times we didn't spend time to discuss our solutions
Lectures were great. I suppose you can still feel whenever the lecture is taught by somebody with and without a substantial teaching experience. I'd like to spend a little bit more time to comprehend everything. As a meta-comment, it might be useful to split all the lectures in two days between two speakers. For example, the first part of the day is dedicated to the old material from the previous day, while the second part of the day covers the new lecture material. The idea is to allow people solve exercises a bit more, think of some questions and ask them the next day.
i'm not sure, but it would be cool to show/include more AI safety relevance; i felt that we basically just did the same math as yesterday in a slightly different setting and it felt a little bit pointless (<- that's a too strong word actually) to me?
There was quite a bit of overlap in concepts between yesterday and today.
As I suggested on Day 1, I firmly believe that a series of two-week "Iliad workshops" or schools on each of the topics covered in the Intensive would be extremely valuable as a whole.
Put the material online the day before. The exercises could be a little more wordy as I had to deduce a few things that could have just been written down.
- sharing the slides (incl. the handout version) and the Colab link before the lecture, instead of only via the QR code in the slides - the units felt a bit disconnected, a short recap tying them together at the end would help
It would be nice if the content and presentation of this day was more synced with the previous one. In many places, explicit comparisons could have been made as to how e.g. the formulas relate to those we saw yesterday, and I expect that this would have cleared up a lot of initial confusion and make the lecture easier to understand overall.
I enjoyed the content, the only change I would make is spending more time on modeern world models and why they work so well (e.g. what we know about "good" latent spaces for generalization and extrapolation etc), rather than the technicalities of epsilon-transducers - only given that the day was titled "world models".
I thought this was very similar to the module on computational mechanics, and it felt a bit repetitive without adding much insight. I thought the exercises were too easy.
There was some overlap with the previous lecture, I think this could be improved freeing up space for more material. The later part of the lecture was very interesting, as well as the introduction.
I wonder if this connects with other aspects of alignment, such as training dynamics or physics of deep learning. It connected with the topics of yesterday and was useful that they were built on top, but not sure what the advantages and things that the worldmodel approach miss are.
I would prefer if the teaching days were restructured. I personally find that having a single day on a topic is a bit too much of an intense burst. I would prefer if the day was cut into three parts, each with a different speaker. That way every speaker would be given the same time but it'd be spread out over 3 days. This would help with 1. Repetition: We would return to topics several times, helping with retention 2. Focus: Adding diversity to the day would help keep stuff interesting, especially towards the end of a long day.
Begin with a short summary of the previous day, and perhaps give more intuition for what we are trying to achieve
less today specific but about the entire week. we tend to have discussions every day about how the material of that day connects to ai safety and there seems to be a lot of confusion around it. like for example, there seems to be the somewhat implicit idea that finding the believe state in a transformer helps ai safety by understanding better how they represent data. but it would be very super cool to hear every lecturer explicitly explain why they think this is the most impactful way forward to make ai safer. currently it often seems a bit like research is blindly pursuing several directions that could maybe be relevant to ai safety. concretely I would wish for every lecturer to at the start motivate how this fits into ai safety and motivate their field from there, then present their field, and then at the end tie things back into ai safety and outline what they hope can be shown eventually and explain why this would help ai safety a lot. I wold also love for the lectures to explain why they have chosen to work on this particular field rather than other fields and how confident they are that this very field is the right one or if they're also somewhat unsure about which field could be impactful
I felt like we spent a lot of time today repeating the same material as yesterday, with slightly different notation and very different vocabulary. I didn't really see the point of introducing all this language for obvious-ish concepts until we got to the Factored Representations paper. So perhaps foreshadowing the relevance to LLMs and AI safety earlier in the day, and spending more time elaborating on this material, would've been better for me. Perhaps a few examples illustrating whichever definitions were necessary would've been good too. But I also had a headache all day, so possibly my outputs on this feedback form are not causal consequences of my perceptions or predictions. Please take this feedback with appropriate salt.
Just some notational inconsistencies. It was a great day.
Add corrections to the exercises in appendix
I think some diagrams were difficult to understand. I would have enjoyed discussing J-lens in more detail (also focusing on the actual maths behind J-lens) - I think the maths is kind of unprincipled and makes me question the validity of the claims made in Anthropic's paper. I would have loved discussing it at Iliad.
I thought it was a bit of a mumble jumble. We covered generic "world models", then went to computational-mechanic flavoured stuff and finally did a bit of a lit review on metacognition work. I don't think this worked out. The content has to be better designed and organised.
the lecture felt scattershot, unfocused and lacked rigorous definitions (which IMO are essential to understand the concepts covered today, by unpacking them and giving concrete examples). There were several errors in the slides that hindered understanding. The connection with yesterday's materials (computational mechanics by Xavier) was made but not as well as it should IMO. Also I don't see why world models were necessary for the lecture.
Last 2 presentations were too fast and I would have preferred a more explicit link between the theory and the applications (last part of the day)
Some more explanation on how can human contribute to these research topics competing with LLM. Why we still need human for these tasks, and in what sub-topic would human-participation would be especially valuable.
Slides numbering. Also, I think we should normalize the "Ohmmmm..." introduced by Julian so that we can finish breaks with it more smoothly. MSP was not defined before giving an exercise. Exercises were way too abstract in the beginning, but did improve significantly quite soon. The material itself resembled yesterday but felt much more abstract conceptually. The intro lecture had all this notorious RL formulae that are hard to interpret on the fly, especially without examples that map what formulae convey with each symbol. Part II with transducer for me was hard to follow because despite great examples given by Fernando throughout it felt very abstract and disconnected from reality still. It might well be that my brain is just not made for this kind of abstraction specifically though. Part III was much more computational and accessible to me, but some concepts (MSP) did not look too well defined to wrap my head around them. Finally, the produced fractals, as a result, did not click. Yesterday is a good comparison -- I was in awe. Today, however, I was utterly lost on what makes this promising.
nothing
Very minor thing: would be helpful to make the terminology between yesterday on computational mechanics and today consistent (belief states vs MSP; i don't thin they were explicitly called \epsilon-machines yesterday?) Otherwise, I thought today's lecture worked really well with yesterday's lecture.

Changed my mind11

world models are the next big thing!
For a long time I've seen work on agents as either theoretical that narrows to tabular RL (which is kinda boring) or purely empirical. The lecture and connections to computational mechanics as well as discussions on world models helped me reassess some of these views and think of new connections, especially to tensor computations I've been working on for a while.
I updated my definition of a world model.
The J-Lens!
I found the world model material very interesting and I plan to look into this material further.
n/a
it seems there is some real depth in this research agenda of computational mechanics. compared to mech interp I like the idea of building up synthetic data as ground truth and studying whether an LLM represents this data in the optimal way which it seems to do. I am still somewhat confused about a lot of the concepts covered today but it feels worthwhile digging deeper
I am more confident that this field has impact on AI-alignment and safety. While finding it fascinating, I was afraid it was too theoretical to be useful in the short term.
It motivated me to check some frontier papers. Also discovered some AI programs throughsome other participants.
This side of comp mech did not convince me to be worth pursuing, most likely because, as Fernando mentioned, it's hard for the bottom (Transducers) and the top (Metacognition / personas) to meet in the middle, at least yet.
I believe this question should be written 'Did you change your mind about anything today?' or 'How did your mind change with regards to a certain topic' or something. Anyway no idea still figuring it out

Other thoughts3

n/a
Today's schedule is a bit tight
To make it more general: I think we should normalize the "Ohmmmm..." introduced by Julian so that we can finish breaks with it more smoothly. As otherwise lecturers need to be more loud to call people in, and that is not as effective

August 2026 Thu 13 Aug Fernando Rosas 28 responses energy 4.0/5

Called “World Modeling + Transducers” that cohort.

Rated against what each person expected. 0 is as expected.

Overall +0.75 better than expected

Content +0.79 better than expected

Clarity +0.71 better than expected

Motivation +0.36 better than expected

Exercises +0.36 better than expected

Readings +1.25 better than expected

Other activities +1.00 better than expected

Pace -0.21 about right

Difficulty -0.11 about right

Novelty +0.68 mostly new

Read what they wrote42

Worked well13

time for exercises and lectures was good
The structure of the course: i.e how long the talk was and the length of exercises. Things fit together nicely.
I liked starting with the motivation and then getting into the details.
The first set of exercises and the second lecture.
Both talks (Alec & Fernando) where excellent
Loved the balance of theory and coding exercises.
Now I can finally read the fractals in Simplex's papers!
Alec and Fernando are great teachers!
Exercises worked great. All Lectures were at an appropriate pace.
Alec was very good.
The first class in the afternoon was the best we had until now, despite problems
The world models introduction was helpful and fun, the transducers section from the second part was very helpful, and the sneak peek of Alec's research results was very interesting
I think the excercises were actualy really helpful

Would improve17

MSP was not defined in the lecture notes, but it was what we were asked to draw in the exercises. Might be good to do a similar example before the exercises.
Not much. Perhaps more discussion about the relation to the previous day.
I would like a bit more open-ended research-style questions. Those motivate me more than computations. Also there could be a bit more said about probing world models as I think that is quite important.
I think the first lecture should have more content. The second set of exercises were not very helpful-there was strong overlap with yesterday's Jupyter notebooks and also were super computational (in an unenlightening way). Finally, I thought of a suggestion for yesterday-have the lectures follow the notebook and connect the notebook's terms to the lecture/AI safety. I found the exercises to be very tractable mathematically, but not connected to AI safety.
The exercises could have been a bit more interesting I think - a bit more mathy and/or a bit more conceptual
Slightly more challenging exercises.
One of the best teaching and learning experience this week. I like the high level overview in the morning and the more hands on approach during the day.
yeah I think the work was insane. It only makes sense why alignment is incredibly difficult and a theoretic perspective is even harder
I think it was great to see a lot of the research in practice and how the objects we learned about in the morning was relevant to results we were showed
Use the gong more often, I was late to the lecture because I was waiting for the gong.
More time for exercises is always better. Maybe useless critique because I'd guess for most people this was not a problem: I like to be very detailed with my probability (for ex., use Kolmogorov extension theorem to actually make sure giving prob of any word suffices to describe uniquely prob over bi infinite sequences), so the class was great in the temptative formalization of stuff, probably the best we had until now BUT the way physicists/not mathematicians handle this transducer thing is very confusing, of course if you draw an HMM it's easy to compute stuff, but the foundations from the class were a little messy. For example, we define an interface anticipation free iff p(y_{:t}|x_{0:})= p(y_{:t}|x_{0:t}) and this is confusing because then if I define my process to be Y_t = X_{t+1}, how do I compute p(y_{:t}|x_{0:t})? Answer: you actually can't unless you put a prior over inputs, but this notation of using "conditional symbol" when in practice we're actually having a markov kernel and then anticipation free means something a little bit different was kind of confusing. Not too big of a deal for most people I guess, I just solved this problem asking claude to formalize every little piece for me so things became clear, but I do wonder (and this is also the case for the other days, less perceptible for me because I had more context): if the plan of ILIAD is to bring more mathematicians into alignment through the intensive, then perhaps the classes should be more careful to not be hand wavyy, this keeps applied cs, applied physicists, etc. happy but myself - that am not even a complete mathematician - got a little sad with the cumulative of hand wavyyness with stuff, maybe other people feel similar (to be clear, there is a bunch more relevant stuff than my tiny prob detail that is hand wavyy and not precise, it's just that I'm being able to solve the formalizations through previous contexts + used to translate physicists/other areas stuff to actual math, but at some point I won't have this context). To be clear I don't mean that I want everything super formalized and complete, precisely because this is sort of impossible in a new field, I'm used to working on under defined problems for example, but I think doing the work of separating exactly what is under defined and what is clear and communicating is kind of important and somewhat the major benefit we get from classes - structured learning as opposed to reading papers (think of someone using the unknown hypothesis/theorems as black boxes, but they're isolated)
I think we spent too much time on the concept of world models in deep learning which even according to the presenter does not have an agreed upon definition. Also, the afternoon content seemed pretty similar to the content from the previous day but with new notation etc. While new to me I left without knowing why it was needed vs the HMMs from the earlier day.
I would love to see what this framework was being used for!
great lecture! Maybe it would be potentially helpful to include more open questions?
I was already comfortable with automata / Markov chains / Bayes' rule. For me, the entire challenge of the exercises was in parsing the notation, and getting code to run. (This doesn't necessarily mean the curriculum needs to be changed though, since other people don't have my background.) As I think I've said every day so far, I wish the connection between the math and AI alignment would be more explicitly explained.
This (transducers) seemed very redundant given yesterday (HMMs). It's just HMMs where you can switch the HMM transition probabilities used each step based on the input. The concepts and the exercises could have been gotten through much faster.
Today was a really good session. The one thing that would have been more helpful would have been a slide with all the definitions introduced in bold just before the excercises but that's it.

Changed my mind7

I can see the hype about world models.
I think this is a very interesting direction.
No change
Still kind of unclear of what the world model is supposed to be. Seems like there is a big gap between the theory and the toy examples to real life problems. Would love to investigate how this scales.
Some of the works are quite similar to my on-going work in Multi-agent RL. Looking for specific ways to connect the language.
Amazing that these toy models look very promising
I came out finally understanding belief state geometry fully, and it cleared up a few misunderstandings about it that I had absorbed so that was really helpful. I also came out with some more ideas of computational experiments I want to try out now.

Other thoughts5

would be useful to make sure the gong is always sounded before a session starts
Fyi - I will be missing a few hours tomorrow due to a medical situation. Nothing I need help with, not an emergency, but something that shouldn't wait
Great that today there were group exercises.
Sorry, I forgot to submit this yesterday!
N/A

April 2026 Fri 24 Apr Fernando Rosas 11 responses energy 3.5/5

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.1 of 5

Teaching and format 4.0 of 5

Difficulty match 3.0 of 5

Energy 3.5 of 5

Read what they wrote32

Worked well10

Good presentations
I really liked this slightly more generous break schedule, combined with the earlier stopping time: I felt present and engaged almost until the end (and with just a slight slump at the end). Fernando has a natural way of drawing his audience into dialogue with him and sharing in simple words what he feels/intuits about the material before and beyond the provable theorems it contains, which also helps a lot.
The content was interesting and the lecturing was good. I also appreciated that neural networks were actually mentioned today, instead of just abstract reasoning about math
The pace was relaxed. This is not always a desirable thing, but today I appreciated it.
The content was really interesting. It was nice to have long breaks in between lectures to actually rest my brain. The multidisciplinary aspect of the talk made it very engaging.
I found today's lectures to do the best job at connecting different earlier topics. This is perhaps not surprising given that we have a quite large (and still increasing) ratio of covered curriculum to draw references from, but Fernando worked well on citing or pointing to relevant topics (as well as had many to begin with).
Lectures in the afternoon on transductions, hierarchical emergence etc were so interesting and well presented. I really liked having today so close to abstractions day.
Enjoyed the lectures, found these a reasonable overview of some nice world modelling ideas and the area
A good intro to the idea and ideals of world models, and current methods for course graining decision processes.
This is the sweet spot for me between philosophically interesting and being plausibly empirically testable (even if we can only test in contrived environments for now).

What they took away11

The concept of computationally closed coarse-graining
It was fun to talk to Fernando
Negative probabilities! (Or, negative transitions, latent dynamics that are not probabilities.) I hunger for another way to interpret them.
Learned what \epsilon machines were! Also, the relationship between symmetries and hierarchical coarse grainings was interesting.
I learned another way of formalizing abstractions/coarse-graining, which could be useful for comparison with the Computational Mechanics and Abstractions modules.
Understanding JEPA and V-JEPA as I have heard a lot about it in the news and here it was very well explained. The generative simulators Sora and Genie also were really interesting.
The epsilon-machine-factorization-computationally-complete-levels correspondence (i'm just hoping i'm getting the terminology right, my attention span was occupied with listening and didn't take notes...)
Maybe one of the results about unique smallest transductors, or maybe the initial intuition for the hierarchical stuff.
having the 6 axes of world models as a basic guide, seeing the approaches to the diagrams for transducers and the corresponding results for these, clear abstractions for the justification of coarse graining in the computational and information settings, and nice ideas on using symmetry to generate these
It was a good reminder to think about scaling and course graining more broadly in training neural networks!
The last lecture was extremely useful about hierarchical emergence and the connection with computational mechanics.

Would improve11

There were some connections with actual RL/AI systems, but I think the lectures would benefit from having more of that. Also, the initial presentation about the concept of world model was not super clear for me / maybe going in too many different directions or presenting too many different versions of world models.
Maybe more exercises instead of only presentations/ more discussions?
I didn’t like the first slide deck generated by Claude, after Fernando expressed his own lack of confidence in it. I didn’t feel sure whether I could trust any individual statement or formula I was reading on screen to be true, without asking for confirmation.
More solo/group work to parse through things on our own might have been nice.
The first (AI-generated) slide deck could have been more visually engaging. The second presentation was much better in this regard. The content also felt somewhat similar to the Computational Mechanics day; maybe parts could be merged, or perhaps the differences could be highlighted more emphatically.
The speaker talked a lot about their research and how it connected or developed these concepts and I would have liked to hear more about how you research such concepts and what it may look like.
At some points i found the lecture somewhat too fast-paced to think of interesting, informative questions, of which there were fewer today than on many other days. (Looking back, this also applies to Daniel's and David's days.) I'm not sure to what extent this is a conscious design choice, to prevent being stuck at possibly unimportant details, or perhaps due to the day schedule being about 1/5 shorter.
Morning lecture on world models was a bit too AI for me. Just lists and sub lists without a ton of substance and definitely not at the level of quality of Fernando’s other lectures (including from the RL day)
Probably a few more practical (either maths or coding) exercises would help (particularly if the day is longer than until 4.30)
It would have been nice to include some more interactions - eg discussions in groups to come up with good criteria (or examples) of world models, or exercises involving course graining.
I approve of getting Claude to generate presentations/do research etc. and it’s good to see presenters do that and say they’ve done that (to establish new norms) but the slides should be checked and reviewed before the lecture is given and this wasn’t the case in the opening lecture today.

Worst-Case Interpretability 1 run 7 responses, 21 written

for pasting into Claude

April 2026 Sat 25 Apr Louis Jaburi 7 responses energy 2.9/5

Called “Limits of Interpretability” that cohort.

Rated 1 to 5 in the absolute. Not on an axis with the signed means above.

Content 4.0 of 5

Teaching and format 3.6 of 5

Difficulty match 3.0 of 5

Energy 2.9 of 5

Read what they wrote21

Worked well7

It was useful to think about how we could be fooling ourselves or not using our time well regarding current interpretability methods.
Great morning lecture and Python notebook to explore/internalize it.
Overall case for finding a better mechanistic approach to alignment than traditional mech interp was really well presented I hope it was compelling to the people who hadn't been privy to this idea before.
Good scope on lecture, exercises, and readings.
The presentations were great, and the time balance with the notebook exercise and reading session was good.
The drawing back to the research the lecturer had done and is doing was really interesting as we could see how this was researched in practice.
The amounts of time allotted for the exercises and the readings seemed appropriate.

What they took away7

The arguments for caring about faithfulness and compactness were useful and interesting to learn about.
Exposure to the idea of formal interpretability certificates. Even though the way they are currently implemented seems hopeless, it’s an interesting concept to aim for in other approaches.
Doing some basic compact proofs I think this was pretty useful for my intuition (although it seemed to confuse some other people who couldn't immediately imagine how a proof like this could incorporate facts about the trained model's weights)
Balance between cost of proof and strength of bounds, interp is doomed
That some people are actually proving bounds for neural networks, but that this is hard to scale. And that there are informal notions of scaling this by weakening from proofs to 'heuristic arguments'.
The idea of compact proofs as a whole is a interesting framework.
There are backdoors that are NP-hard to find; “interpretability is doomed.”

Would improve7

I think the fact that there is no existing consensus on what counts as a heuristic argument could have been established earlier.
More could be packed into this day. Perhaps other exercises in mech Interp (and its failure?) to contrast with the proof approach. It’s really enough content for just half a day.
Day was obviously a bit short we ended hours early
A few more research directions/approaches and open problems outside ARC (if possible)
The exercise notebook contains coding exercises in the form of cells with "Solution goes here", but the solution is immediately visible in the next cell (instead of hidden in e.g. a drop-down). However, I think most value comes from the math proofs, so this is not a big problem. In fact, I think the coding exercises could be removed as they are just implementation details.
Maybe seeing some of the code that was used to make the lecturer's paper would have been interesting as a concrete example of this research in practice.
I have heard comedians say that if a comedian seems nervous, then the audience feels nervous on their behalf and can’t laugh; as a theatre critic reviewing four weeks of one-man shows, I would say that Louis doesn’t seem to enjoy or play in the role of presenter as much as our other teachers do. He crosses his arms and flashes infrequent smiles. He had much less material prepared for this topic than his previous day’s.

Reading the bars

Every bar is the answers themselves, laid on one axis with the middle of the scale at the centre line. Left of centre is worse, right of centre better, and the widths are comparable from row to row.

much below below as expected above much above

Pace and difficulty are a direction, not a quality: the centre is the target and both ends are a drift away from it.

on target one step off two steps off

A row says n only where fewer people answered it than answered the day, such as a question about readings on a day that set none.