By cohort

September 2026

November 2026 October 2026 September 2026 August 2026 June 2026 April 2026

Rated against what each person expected. 0 is as expected.

ML Engineering Mon 7 Sep, Julian Schulz 41 responses · energy 3.9/5

Overall +0.05 as expected

Content -0.02 as expected

Clarity -0.07 as expected

Motivation +0.02 as expected

Exercises -0.29 below expected

Readings 0.00 as expected n = 9

Other activities -0.05 as expected n = 21

Pace +0.17 about right

Difficulty -0.32 too easy

Novelty -0.29 mostly review

Read what they wrote77

Worked well27

I enjoyed the lessons before the last hour acceleration.
I really enjoyed Julians talk!
thinking through the design of the adam optimizer together was fun
I liked that the teaching was interspersed with exercises
It was nice to have a bit of time at the beginning of today to meet the other participants.
meeting the other participants and lunch was good, i think this was roughly the main purpose today?
Talking to people was useful and getting to know everyone.
Socializing was great.
The introductions were very nice, I enjoyed chatting with a series of participants one-on-one.
The hands on exercises and the capstone project.
I appreciated block-diagrams and examples. It would have been useful to introduce notation in the presentation deck, instead of having to ask and receive that information orally.
Intro to ML lecture was delivered very well.
Discussions to get to know people.
The networking with prompts
Lecture split with exercise sessions allowed to not get tired thanks to context switching. Also the breaks. The slides were good as well, albeit not sufficient for proper explanations, but that's perhaps too harsh for the overview.
- claude capstone project was very enjoyable, best part of the day - socialising/ ice breakers were effective, met nice people and didn't feel awkward/ intense as some forced ice breakers can be
Good and quick recap of things to get us warmed up
Stopping frequently during lectures for exercises and relaxation.
The short interactive parts are good, both to break up the flow, but also to get people thinking on their own, which always helps learning. The conversations during the first half of the morning were very good for getting to know people. Different formats could be beneficial to have more varied interactions (i.e. discussions in smaller groups (of 3/4/5 ppl)), and these might be better matches for some of the questions/prompts given, but these would let you get to know more people faster (which becomes more important the bigger your cohorts become).
I enjoyed the get-to-know each other in the beginning. I really enjoyed the ML fundamentals lecture, I think it was very well prepared and understandable and engaging.
The questions from the audience and answers from the lecturer were great.
Open questions to class, the Claude sessions
It was good that we spent some time introducing ourselves to each other. I gained a lot more clarity on the background of other participants, and their motivations for the intensive/fellowship.
Having a small group to learn the basics of Claude Code
I liked the exercises format. Both the claude code ones and the notebook ones.
The claude code exercises were good, although not described a priori very well.
I thought Adam explained a lot of the features of Claude Code and coding agents really well, and I got a lot of clarity on a few things I didn’t know as much about initially.

Would improve31

Better explanations of technical concepts (such as transformer architecture), and I would have enjoyed a deeper (perhaps more formal) approach when it comes to teaching in general. I think that there was not a great time control. No time for solving the exercises, and I think the socializing exercises took too much time.
Sometimes when the group was split, it was unclear what the options were regarding the possible choices. Discussion games at the start could've been improved a bit, but I think the main restriction was the room shape. Would've been nice to mix a bit more, and maybe having another in the future where you talk to other people would be useful.
maybe scale down the introduction part in the morning and split the intro to ML into 2 sessions, by the end of the ML intro I couldn't focus on the content which was still very interesting just due
Adam was great, but I don't think the "claude basics" activity was super useful. I felt like it was an intro to vide coding, but I think it would've been useful to go from the very start to the paper replication and getting hints as to how this could be done efficiently. Learning about worktrees whilst doing this project would've been more valuable.
I found the today's content very helpful but I felt like it went too fast. I think I would've appreciated if the same content was instead spread over two days. As someone who had a low level understanding of about half the content before hand I would've liked a bit more of a chance to work through exercises and have the instructors roaming around for questions.
I think choosing between ML fundamentals + Claude Code intro is not the most ideal - naturally I wanted to cover everything so chose the former which left less time for the latter. I wonder if the first half of the day (intro stuff) could be shortened, or start an hour earlier, to fit in both?
We were given a couple of choices today for what material we could learn. I think it would have been helpful to know in advance what was going to be covered, as there was a bit of a rush to decide (and without really know what would have been done in each session). I did the ML basics in the afternoon, but in hindsight, I think intro to Claude would have been more useful for me. My answers below reflect the fact that I did the ML basics, parts of which felt fairly familiar. I think it might just be difficult for people to choose, so I'm wondering if it'd be possible to compress the ML basics / intro to Claude so everyone gets to do both? I felt the "try to get Claude to replicate the paper" exercise was a bit less helpful. Perhaps it could have been helpful to have a bit more guidance on this part (what would be the "best practices" way of doing this)?
If you put formulas on slides explain the terms (what is the sum over, what is y and what is y hat). I would prefer to do deeper reading over a rather superficial lecture. Smaller group size would help for that.
The Reinforcement learning section was somewhat too quick, I would have spent some more time on that.
It felt that the ML/AI intro was not too well structured. Arguably, that's a common problem for many such intro classes. E.g. RL was surely too math-y compared to the beginning, and perhaps not as relevant in fact to RLHF (or the model lifecycle is not relevant to "intro to ML"). I also felt that there is a real discrepancy in participants' knowledge (it was to fast and superficial for those who didn't know it before; boring for those who already knew), so it seems reasonable to run a small test in the morning of the first day and then split into groups where the intro would be more targeted. Also, it looked like the diagrams in the presentation were a bit rough (in my experience it was similar to unpolished tikz code generated by a model). Finally, it may be worthy to emphasize that this is not academic setting anymore, so people should not be afraid to give wrong answers -- at times there was a bit of awkward silence. Socializing part was cool, but the room is too small for 40 people talking simultaneously, so it was overwhelming.
I'm not sure if I got very much out of the Claude Code session. I learned a little more about how it works, and a few handy tips, but nothing that I expect to really transform the way I use it. This may be my fault for choosing said session; I already use Claude Code quite a lot and was at least somewhat aware of most of the features we covered. So my only pedagogical suggestion is probably the session could've been a bit shorter. Working through the exercises was mostly a matter of waiting while Claude ran, only a little bit of the time was spent trying new things.
Claude code exercises are poorly explained and assume a lot of prior experience using claude code.
The session in the morning designed to get to know each other was way too long in my opinion. After the session, there was also a break and lunch which achieve the same objective. Because we spent all morning on this, the afternoon session was a bit rushed. I would try to include a part of the afternoon session into the morning.
It's better for Claude Code exercises to have more structure and well-defined goals to verify your abilities upon. Another way to approach it is to build it around designing skills and workflows yourself. Yet an exercise should incentivize the participants to do it.
Voice projection. Unfortunately, it was very difficult to hear instructions when organising the introduction rounds, or to understand when there were breaks. Regarding the lecture: a poll among participants well before the course would have saved awkward hand-raising moments to decide what to cover, or how to distribute groups and in which rooms.
I would not do the 'intro to claude code' vs 'intro to ML' split - I think it would be better if everyone does the same thing so it doesn't feel like you are missing out on an option.
Pace could be a bit slower. Temperature in the room is too low.
Improve the pedagogical value of the Claude Code exercises. I didn't feel like I knew what I was supposed to do. Most of it was guess work and hoping things would work. The paper replication is the same. Some guidance would have been interesting instead of simply saying "reproduce figure 3 in this paper". The use of the research project template itself was not well introduced. Slow it down, narrow the scope and pass a cleaner message.
While I appreciate that this intensive needs to cover a lot of (rapidly evolving content), I think the speed at which the fundamentals were covered clearly divides our cohort into those who are more theoretically-oriented with almost no AI or coding experience and those that have used AI tools.
Define the variables in all equations or include on slides
I think I was missing a clearer vision of what will be taught in the fours weeks, this week, and today in particular, how this fits within Iliads vision, what we want to take away from this etc. With the afternoon exercises on agents, I think I would have benefited from a sort of best/good practice for agentic coding. the exercises were nice to get us working straight away, but it was often only through questions or confusions that better workflows became clear to me
Desk placements (happy to help with moving them around!) into more of a school / uni structure (facing the lecturer). Clearer outline of what we would be missing out on if choosing to attend one session rather than the other / choose one exercise rather than the other. Ideally to announce such choices beforehand so we can say skim over the materials and decide in a more improved manner [Generated by my FOMO]. Also the air conditioning over the tables closer to the door make it chilly, and not just if you are under it -- I would switch it off for the afternoon sessions.
- would be nicer to have lecture notes, especially ones that are clear. since this is an earlier program, it makes sense this doesn't exist. suggestion: look for volunteer scribes? you can ask for 2 volunteers per day and get claude to smush it together (although i would put high emphasis on creating lecture notes which don't read as ai generated, this is very off putting) - similarly, it would be nice to sort of do a pass over any slides / material. i think using models to create materials is efficient and fine, but one should do a pass over it or use it to create a skeleton as reading something that reads as "claudish" can be similarly off putting
Introduce motivation of something with pictures during the slides. Sometimes it was not clear. Rest was great. Especially the discussions during the lecture and the exercises.
Simply too little time. I think it's worth extending to get everyone up to speed. This naturally depends on the coming days and in what way they rely on deeper understanding of today's material. Also, the people who most need a break are (generally speaking) also the people who would most benefit from the excercises, i.e. they are the people with the least prior experience. Please keep this in mind when offering for people to solve the excercises during the breaks.
We were given a choice between different exercises/different lectures (either ML fundamentals or Claude Code, and then either Claude Code capstone project or exercises). I found it difficult to choose. I think I didn't have enough information to make an informed decision. It would have been useful to know more precisely what each of the options would entail beforehand!
The introductions parts were working fine without suggestions on what to talk about with each other. Otherwise everything perfect.
Perhaps a more targeted course (i.e. covering less topics), and have people read the prereq. material one week earlier! The Claude exercises were fun, but there were some small API topup issues
More time for exercises. But maybe this will improve in the coming days, because this was just the first day.
The lecture for the ML basics lecture was too rapid for me. I understood what I already knew, and not the concepts I didn't know. Maybe less time for people to try and come up with various gradient descent algorithms from scratch. There are lots of reasonable answers, hard to converge on momentum etc.
I worked through the CC exercises with Adam. While I appreciate that the exercises were a substrate for learning about CC, it felt a bit strange to work through knowing that frontier models can probably one-shot these exercises with no need for e.g. subagents. It might have been cool for the exercises to really show the power of these techniques if possible.

Changed my mind14

Being the first day, got to know other points of view about AI safety, from other participants. Discovered other AI safety institutions and interesting career paths.
n/a
I was expecting today to learn in depth the transformer architecture. Instead we spent a lot of time on basics. I also expected to work on the exercises and set up claude code with vs code before but there were other necessary setups eg GitHub that we were not informed of in advance. Overall I would have learnt more by working through the transformer material on my own and spent rest of the day on the exercises.
n/a
Claude Code in VSCode is a bit clunky
No
1. Gradient descent was improved heuristically rather than mathematically (Newton's method) or physically (energy-based approaches which give smth similar to Newton). 2. That is sometimes does happen that an input from the engineer leads to an improvement (RoPE giving better results than basic encoding) -- which is not how it usually works.
n/a
Not yet
Yesterday night I was feeling very anxious and feeling like an impostor, and I thought the atmosphere at Iliad would be much more competitive, I was happy to find out that I needn't have worried about that!
Attention as a mechanism is more complex than I thought.
Not really yet!
I should get familiar with coding agents
N/A

Other thoughts5

no
n/a
1. Do I need to say in advance if I need to take a call at 17:00? Can I book the box in the Iliad building ? 2. I cannot attend Friday 11th and need to leave at 17:45 on Thursday.
Ad question on exercises: I only really did one exercise (pair programming with someone) - implementing the SGD update. I thought that was pretty easy but I actually learned that you cannot do self.params = self.params ... so that little exercise was very helpful!
I will have a doctor’s appointment on the 14th to forewarn I may not be able to attend in the morning.

Mechanistic Interpretability Tue 8 Sep, Julian Schulz 41 responses · energy 3.7/5

Overall +0.27 better than expected

Content +0.27 better than expected

Clarity -0.27 below expected

Motivation +0.27 better than expected

Exercises -0.15 as expected

Readings -0.25 below expected n = 16

Other activities -0.20 as expected n = 30

Pace +0.68 too fast

Difficulty +0.54 too hard

Novelty +0.63 mostly new

Read what they wrote78

Worked well22

I think the lecture was great, I liked that there were questions to the audience, and I also love the questions from the audience, they are often very thought-provoking. I think the lecturer does a great job answering questions and I also respect his epistemic honesty. And he is enthusiastic which makes it easier to keep paying attention
discussion with peers helped clarify concepts
I really enjoyed the overview of Mechinterp - It was really helpful for me to get a good overview of the area so I can go and learn more about what i'm interested in.
I think the explanations of the topics, and then going on to neuronpedia to actually see it in real models was helpful. I also thought that the exercise of thinking about the induction circuits was helpful
Some more intuitional explanations, motivations, discussions with pears
playing around with Neuronpedia was good to show us how poorly mech interp techniques can work. I didn't find any of the techniques particularly convincing, and maybe that's a good thing.
Julian is a very good lecturer and that helped to understand new and more complicated topics
The initial discussion was well motivated, and examples given were useful.
The first half of the day established the basis of the field very well .
It was a good intro to Mech Interp, covering a broad range of ideas, older and newer. It’ll definitely help anyone choosing to learn more about any topic in mech interp.
The mixing of theory and practice. The exercises were quite interactive, that helped switching. The slides were quite comprehensive and informative, with references etc. Very good job with the slides I'd say
It was easy to find the source articles on the slides
I liked the transformer circuit thinking exercise
Some small exercises with coding part were great. It's valuable to at least sometimes do them yourself without deferring everything to coding agents. Discussions were great as well.
Doing the exercise about the attention with key, queue value adn hearing the explanation of a colleague.
Interactivity.
The coding exercises in pairs were really useful. The fact that we can spend a big chunk of time asking questions is really good too.
the pair programming exercise on logitlens, understanding circuits through concrete examples and trying to understand how queries and keys are combined from different tokens and how positional encoding of the tokens play a role here.
teaching and slides were great!
The teaching.
I'd previously seen most of the material from auditing the ARENA course, but working through the exercise on K-composition vs Q-composition clarified things for me a lot, so that was good.
The pair coding exercise was the most helpful part of the day. It was very nice.

Would improve32

I would have preferred a slightly more rigorous, in depth coverage of fewer topics. Having said that, I think the lecture was great given how much material was covered
I think the pace was a bit too fast today, I think it's just too much material to fit in one day. I was already familiar with many of the topics covered today, but I imagine for someone without that background it must have been quite difficult to follow.
I think more time for the exercises would be nice.
Great content - just felt slightly rushed for 1 day; I think 2 mechinterp days are probably worth it given how central the field is (vs. some of the more esoteric areas covered).
Overall, I thought that the day was well structured and well done. Pretty small thing, but I think I would've enjoyed spending a bit more time on the fuzzy interp stuff, and perhaps slightly less on the early CNN stuff (though I suppose this was already quite short).
Pictures on slides are small and hard to see, but I pulled the presentation on computer so that helped
Seems like reducing the number of slides from 140 to maybe <100 might be a good idea, and then spend more time explaining the most important results. Some parts went way too fast. Usually, if you spend 30 seconds on a slide, its best to remove it. I find the K and Q-composition circuits are better understood when you make each component appear sequentially instead of all at once. You can also draw it on the board and describe what each component is doing. The Q-composition exercise was not very clear.
The material is very interesting, but perhaps too compressed, I would reduce the material (especially on some of the latter part of the lecture) and leave space for a longer coding session! I find that coding the algorithm yourself helps a lot with understanding.
This was a lot of material for one day. I'd split it in two.
It would be nice if the lecturer repeat the questions because you can't hear them well if you are in a corner of the room
As a broad survey it seemed good to me, although of course I probably don't know about anything you chose to leave out. There were a couple occasions where I struggled to follow this "why is it done this way"/"why does this property matter", so possibly there is room for pedagogical improvement in terms of how to introduce and motivate the core concepts? Or possibly this is irreducible complexity and one must start somewhere. Also, it felt like there was very little time for any exercises today. This may be because we asked too many questions during lecture, so perhaps a worthwhile tradeoff.
I think I would skew the content more towards fundamental concepts such as feature and circuits so that they are explained in more details.
Like yesterday, the time is simply too short. This is worse than yesterday as we had a full day of teaching.
1. The position of the laptop that connects to the monitor ends up forcing the lecturer to stay in front of the slides, covering them. The laptop stand should be moved to the side, to the right most likely (the whiteboard on the right can be moved even more to the right as well). 2. Needed more mixing up theory and practice. The interludes with neuropedia were great and gave an opportunity to switch the brain from theory to practice. 2.5. I felt lost at the long SAE section where we did go in depth of too many approaches. Perhaps a brief overview of each would be enough, as otherwise gives mental overload. 3. J-lens wasn't represented on the slides, only mentioned briefly in the end. 4. The breaks should be separated by 45 mins tops. I think a survey / poll on Discord or WhatsApp would be great about the maximum acceptable interval between breaks. We might also create a button for breaks -- during the lecture, if people feel tired they can "push" the button, and once it gets pushed by more than 33% or 50% of the audience, we have to stop in the next 1-2 or 5 mins. 5. To add on the subject of breaks: right now we have 2.5 hours before lunch and 4 hours after to study, which makes it imbalanced and calls for a big break (at least 20 mins) during the afternoon session. 6. In-depth explanations should employ the whiteboard more, or be more example / exercise driven I guess. As it gets literally hand-wavy, but that's perhaps the hardest to improve
Some places around the class are not well suited to see clearly what happens at the board so if possible it would be great to find a disposition that allows more people to sit in front of the teacher.
I would spend some more time on the circuits in transformers and look at some explicit examples on the board. Exactly how the features were used in the IOI or ACDC, the final attribution graph circuit and the motivation did not feel clear. I would also have benefit from some more explicit (low level) examples in the identifying SAE features section, though I'm not sure how feasible that is. I didn't find the exercises useful in understanding the presented material, more for understanding how to code various bits (which is fine if that was the purpose).
Even though I knew a lot from the material today, I find the pace uneven: too much time is spent on simpler or older concepts, while more complicated or/and newer ideas and methods are rather skimmed through. It is not clear what is the pedagogical goal of the day: is it to give a historical perspective of the field? to explain the basics? explain the latest developments? I think that stating a clear priority would help with shaping the material. Going quickly over complicated concepts towards the end of the day increases mental load and disengages from following. It might be beneficial to simply go slowly and drop the volume of the material. Then, even though the excitement is clearly communicated, there's always this thread about uselessness of mechinterp. I understand that it is reasonable to give a realistic assessment of the state of the art, but it demotivates quite a bit.
Non physicist opinion: be more mathematically precise or tell me when you are not attempting to; define all notation on slides (x_i: what is the i?) Include more practical exercises like the transformer thinking exercise The coding exercise in the morning was bad: the google notebook was too slow, and i mostly struggled with syntax (which felt artificial and not helpful to learn) General: 1. Maybe try to screen participants more strongly for caring about AI safety? I feel that some are just here because of trying to ride the general AI hype wave (not crazy sure about this, just an impression). 2. facilitate more socializing (city tour, etc)
A bit more attention to universality and corner cases will be great. I suppose it's fine to introduce most known concepts / approaches and skip some (like cross-coders etc.), and leave them as links for interested people to explore. What can be most valuable is sharing knowledge about some corner cases, limitations and reasons methods might not work instead. I.e. something that is hard to extract and summarize from papers and which requires lecturer's experience and opinion. I might be slightly biased as I'm at least familiar with the field, and I suppose that perspective of someone less familiar will be different.
- Put the course material in one channel, and then in the messages mark the day and the topic. - Be clear what mathematical topics will be explained in detail and if not where to find resources on the ideas. Some of the formulae and proof I could not follow.
Dense slides, hard to follow if you don't already have a reasonable background.
Better than yesterday. Teacher seemed a little bit more prepared.
A lot of material was covered, sometimes at the expense of depth. It was not always clear when results were robust or cherry picked.
I personally need two weeks to do this two day bloc, and it would still feel intense. Perhaps a series called "Iliad Fundamentals" could also be introduced, where each run lasts two weeks and does an intense deep-dive into specific topics covered in the Intensive.
More paired coding exercises. These were super valuable. We also had a decent amount of time to complete this so it didn't feel rushed.
more hands on things and i liked the parts where students are asked to explain something.
- overall very good, would be nicer to have easier access to slides, notes, etc i.e is the https://iliad-team.github.io/iliad-intensive/ website going to be updated? i think something like this which has direct link to readings, slides, exercises, notes would be nice - teaching was very well done
Overall I think Julian did an amazing job of covering a lot of material in a short length of time and the teaching was great. I think the demos could be improved as it was not super clear how to use e.g. Neuronpedia. Perhaps instead of breaking into pairs this could be demo'd to the group as a whole (and still be interactive for people to guide the prompting etc.).
I think the lecture (and lecturing!) were really very good - although I can imagine the pace would have been very difficult for someone not already familiar with the material. I think some of the precise mathematical statements of results could be corrected. Naturally the statements have to be abbreviated for clarity, but for example, the statement on the Compressed Sensing slide (s58) is clearly false as written.
I enjoyed the content we covered today but I felt like we went too fast. Especially during the second half of the day I felt like I was accumulating lost-ness. I really enjoyed the pair coding and would love to have much more time dedicated to exercises (not just the breaks). I feel like my capacity for new content will grow as we go through the month so I think I would benefit from less content, more examples in the first week.
We went really quickly through the slides.
The redirections to neuronpedia were great. It might have helped to include at least one slide explaining what to look for. Same applies to the many graphics (which looked awesome btw).

Changed my mind17

not too interested to mech interp anymore
I have a clearer idea of what is mechanistic interpretability and I find it really interesting as a potential line of research I would like to work in.
I am new to the filed and I found the topic interesting and worth learning about
Mech interp seems less promising than I thought. SAEs seem pretty useless, most features looked uninterpretable.
I realized probes could be a very promising direction to study.
I have a much more detailed understanding of what's up with mech interp now, but I think my overall feelings about it are roughly the same.
Maybe was a bit discouraged that mech interp will be “successful”.
I did note down quite a few insights, but I am spent for today, no tokens left
I have a clearer view of what we call a feature in MechInterp
It might be that I underestimated some method's promise, and i.e. NLAs, if done properly, might be useful overall.
No
I previously had the impression that mech interp was more rigourous than I do now. A lot of techniques seemed to work in some instances and fail in others, without a clear explanation of why.
I am not super convinced on the utility of mech interp for safety but I think it's valuable that someone is pursuing this.
became slightly more pessimistic about mech interp but more about the other parts of ai safety coming up next :)
n/a
I was a little surprised to hear about how unreliable mechanistic interpretability methods are, which makes me a little less likely to consider research in this field as opposed to another subfield of AI safety.
I discovered the sota of Mech interpretability, its limitations and directions. Maybe worth considering for my future career.

Other thoughts7

Today's lecture was much better paced, and pitched at a better level.
nope
I had a hard time finding the slides. It appears one of the two links to slides on this page was broken: https://iliad-team.github.io/iliad-intensive/interpretability/mechanistic-interpretability/ links to both https://raw.githubusercontent.com/iliad-team/iliad-intensive-C.2/main/lectures/output/main.pdf (404 error) and https://iliad-team.github.io/iliad-intensive/downloads/mechanistic-interpretability/mechanistic-interpretability-slides.pdf (loads, but I think was missing some slides vs the lecture)
This is actually such a great experience, I love being here among the peers. Thank you guys for making this happen, I hope it goes smoothly!
it would be nice to have more desk space, especially orient the desks so its a lecture form and they all roughly face the front?
n/a
Apologies for arriving late today! It was an unexpected delay resulting from my train being redirected, but I'll set off earlier from now on to avoid it happening again.

Physics for Deep Learning Wed 9 Sep, Tudor Dimofte 42 responses · energy 3.8/5

Overall +0.62 better than expected n = 39

Content +0.72 better than expected n = 39

Clarity -0.13 as expected n = 39

Motivation +0.18 as expected n = 39

Exercises +0.15 as expected n = 39

Readings +0.47 better than expected n = 38

Other activities 0.00 as expected n = 26

Pace +0.41 too fast n = 39

Difficulty +0.38 too hard n = 39

Novelty +0.72 mostly new n = 39

Read what they wrote62

Worked well20

Tudor was a very clear and engaging lecturer
I found Tudor to be a great teacher!
doing the lab of lazy vs rich models was very fun! Doing the scaling exercises was nice too.
Exercises, lecture before lunch
The presentation -- LaTeX-rendered is a love letter to Physics Academia. The breaks composition -- very balanced, and having a big break in the middle of the afternoon session -- thank you for that. The fact that lectures weren't longer than 1 hour. The exercises. It was a great day overall, I liked it.
I am very grateful that the lecturer answered questions during the breaks. These explanations really really helped my understanding. I also loved discussing the material with other participants. The reading at the end of the day was great. And the discussion was also thought-provoking. I liked the 3rd exercise sheet most.
Clear synchronisation between lectures and exercises. Solutions provided as well.
We had lenghty periods to reread and ponder the materials
I think that the discussions / exercises in between lectures was very helpful (it was nice to have a bit of a breather between lectures)
The reading and the overall picture of the field.
The content is superb. Super interesting! The exercises are good but sometimes I got a bit lost in the bigger pictures, so even if I could do the maths in the exercises it wasn't useful to do them because I didn't get the utility/what it was getting at.
I liked the teaching style. The topic was very interesting.
The exercises were well-chosen.
Pace and amount of material were adequate and not overloading
I thought Tudor was a great lecturer.
More structured breaks, and lecture lengths. Also the pdf of lecture notes and exercises with solutions is super helpful.
- exercises, extra reading, notes (super well put together)
The long break, the different exercise options. The very clear structure of lecture, exercise, break.
I though the lazy vs rich learning dichotomy was very informative.
it's nice there was more actual in-depth content today, but i was lost super quickly (no physics background)

Would improve26

Perhaps take longer time with some of the more difficult concepts like NTK, I found it difficult to follow (without a physics background)
Occasionally, variables weren't explicitly defined on the slides, so stating what they all are more explicitly would be good, especially when notation changes between topics/slides.
I would divide all this day's wroth of content into 2 or 3 days (or even a whole week!), to really go deep and give us actual tools to use from it, as opposed to just having a vague recollection of what happened.
Nothing was a really interesting day
slides are loaded with information that is sometimes glossed over quickly and could gain from more plots to visualize actual research results.
Make the connection between deep learning and physics clearer before moving onto the physics introduction. The physics introduction was a little derailed by people asking questions attempting to connect the (familiar to most) physics to deep learning.
Post lunch the slides were hard to follow
I think notation in the lectures today could be improved. I was getting lost sometimes. Same goes for exercises. The NTK was quite confusing for me and a few people around. That part should be more visual and geometric (just like LR is explained via N-dimensional vector Y being projected onto d-dimensional space of N-dimensional vectors X (d=number of features)). Also some people were in quite a disbelief that this is useful and asking "So what?" and "It's a toy model, how is it generalizable?". A better detour into Physics for how fundamental knowledge was opening the doors to new scientific directions and technological explosions (Quantum Physics, perhaps GTR), as well as giving us profound understanding of things we could hardly define in the past ("features" is one such term).
The material was quite difficult for me. I think it would be helpful if during the lecture we more often drew these connections between the physics theories and neural networks. For example we talk about Ising model, we have these nodes etc, and nodes correspond to X in neural networks. And Y in physics corresponds to Z in neural networks... etc. So to summarize, I was quite lost during the lectures today. But then during the breaks I asked questions and this helped my understanding a lot!
Today was quite nice!
Add more pictures to the presentation, maybe give more intuitions and fewer equations
Content was good, timing was a little suboptimal (limited time for exercises/readings/questions/discussions), I think it would've been nice to have a few more pictures in the slides. For example, there were a lot of questions when the Bayesian learning model was introduced. Here's a picture that I think would've helped me grasp the motivation: A 2D schematic of the parameter space, loss landscape sketched topographically. Upper panel: Widely-scattered cloud of points representing initialization distribution. Lower left panel "SGD": Cloud of points obtained by running SGD for a short time, more concentrated towards minima. Lower right panel, "Bayesian learning model": Cloud of points obtained by sampling e^{-\beta L}, also more concentrated towards minima, hopefully looks similar.
It would have helped to have a bit more details on the slides especially regarding the definition of the notations used
Another great day of content - only limitation was I felt the methods covered could have been slightly better positioned against the wider literature on end-of-learning and learning dynamics.
Potentially, having some option to make notes directly on the slides or project the presentation onto the board could be great, so that the lecturer could derive and explain some things more clearly without hesitation. This is more of a meta comment though.
I would have liked a more basic and foundational introduction to the field, and maybe some mentions to physics for ML apart from statistical physics.
Quite a bunch of regimes and models were introduced and while they were self-contained and clear in themselves, at the end of the day, I still feel a bit fuzzy as to when and how each is applied. Maybe taking like 5 minutes at the end to recap where each fit and what they do well would be useful.
Extremely interesting content which was relatively well delivered. I think the lectures were an adaptation of the content in the paper he shared ("There Will Be a Scientific Theory of Deep Learning"), and he added the necessary context to understand this well. Comments: - The motivation wasn't extremely clear. The motivation is the paper is much better and I think the lectures would benefit a lot from "copying" a bit what they did. - The prerequisities for this day are not really present, eg. I went on a rabbit hole trying to understand the differential geom part of the lecture (which I loved tbf). This can be completely skipped by doing everything in R^n, OR include in the prerequisites the tools used so that we can revise them a bit before. - Overall, there was a lot of content. This is very interesting so I think it would be valuable to have 2 days instead. Also, the slides are too dense. We have a copy of them so extend them and write things fully!!!
Very interesting day. I wish I had a stronger background in statistical physics to understand the classes better.
The teaching and excercises were very good, I have nothing to suggest improving upon!
I think the section on NTKs, while it felt well-motivated and it felt like the material covered was well-designed, was much denser than the rest of the lecture(s) - I'd probably allot more time to this section than we had. I'd say we probably didn't need as long breaks as we had. Also, I didn't really feel the chosen material covered much about phase transitions, although I understood how the Ising model was relevant in other ways.
I enjoyed there being structured lecture notes and exercises I can return to later. I enjoyed today.
very little suggestions apart from improving iliad-team website with more centralised stuff for material. exercises are good, would want more of that. would want more detailed prereqs for the day though, leadup can be hard for non physicists. the notes and readings are great it was just a little fast.
A bit more bigger picture.
I thought the slides could have had more plots on them. A lot of heavy math but not enough qualitative interpretation of it.
maybe: include some option B material for TCS people who are lost on like slide 2

Changed my mind13

All of this was genuinely new to me and I found it super interesting. Will certainly explore and think about these ideas further.
Im still thinking about this
there are more links between physics and machine learning than I expected, but it stays very complicated.
That it is actually more useful and promising that I thought -- it gives us mental models and predictive power for some things. However, it is also much harder that I thought. Also, the paper "There Will Be a Scientific Theory of Deep Learning" underplayed scaling laws -- we do have models for predicting exponents, but how successful they are is a different question.
I understand not that there may be such a thing as a "universal theory of learning mechanics".
Not really; I was hoping for more discussion of how physics of deep learning work relates to safety goals, I don't think my understanding of that picture really changed.
In the end we skimmed through a paper arguing that there will be at some point a solid theory of learning mechanics which I guess would help to consolidate the different efforts thrown into AI alignment so if this is true I think it is quite good (except if it allows even more rapid development of the AI capabilities comparatively).
Maybe because of a bias or a misconception coming from myself, I thought that predicting features of a model at a large scale had to be too unreasonable. However, to my surprise, a physical-like approach was able to be useful to predict some behaviours/laws. I found this particularly inspiring.
I learned a lot about physics of ML. It helped me understand not only physics of ML, but also the inherent concepts of points of view that physics itself is based on.
Yes- I want to have a look at this in more detail.
I didn't really see how the topics we covered today directly impacted AI safety, although I can see how research on these topics could be valuable to anyone working in AI.
A bit of increased skepticism on if the physics toy models are the solution to the AI Safety problem and unsure about how much part of the AI SAfety pie they should have.
It might be possible to theory-model deep learning.

Other thoughts3

n/a
Great teacher. Leveled with us students a lot, was very curious and into what he was teaching! Liked that he went into details a lot.
N/A

Computational Mechanics Thu 10 Sep, Xavier Poncini 41 responses · energy 4.0/5

Overall +0.92 better than expected n = 39

Content +0.67 better than expected n = 39

Clarity +0.51 better than expected n = 39

Motivation +0.10 as expected n = 39

Exercises +0.51 better than expected n = 39

Readings +0.63 better than expected n = 35

Other activities +0.25 better than expected n = 32

Pace +0.03 about right n = 39

Difficulty +0.03 about right n = 39

Novelty +1.05 mostly new n = 39

Read what they wrote65

Worked well23

I really enjoyed the pacing and intuition / explanations of the lecturer
Described above.
The progressivity of the concepts presented, also the article with a short summary and a guide to point out the most important sections to read.
Slides were very clear, and the exercises fun and instructive.
Good explanation of generative models, linear embeddings, sufficient stats; good discussions.
Learning new ideas
It was great.
Long exercise sessions, with exercises not being too hard, so it was possible to get a grasp of all the concepts covered
really enjoyed the python exercises for computing belief states and the quiz (good check). also easy to follow "story" of going from hmms to belief states to the transformer result
Clear build-up of definition.
The answers given by the teacher and conversations with him and other participants
Lectures were well structured. I could clearly follow how the ideas were building up. The readings were relevant, and I was impressed I could understand a technical paper on a completely new topic after only a few hours of lectures.
Exercises
Xavier was the best lecturer so far. Found him very easy to follow.
The clear structure with the timetable, it was paced very well with plenty of time for questions. The exercises were at a good level and very clearly linked to the slides and the paper we read at the end.
The structure announced in advance: a 30 min intro lecture with two follow-up deep-dives. Having an intro lecture really helps. The slide numbering -- makes it easy to do notes and refer later with the questions. The exercises -- really drove the point home and assisted with understanding a lot, as it felt a bit abstract at times. Finishing with reading and discussing a paper was a top notch, just like yesterday with Tudor on Physics of Deep Learning. I love this structure so far.
the exercises are very well chosen. The presenter was very good.
I liked how the day was broken up into lectures and exercises/reading
As opposed the past three days, the material, while advanced, seemed more targeted and far more clearly presented in the slides and the problem sheets. The problem sheets themselves were more helpful than those of the past three days as they were detail-oriented and low-level, catered towards enhancing our understanding of the very basics of the new techniques introduced today. I did not feel overwhelmed by them and came away feeling like I measurably learnt something new. The larger amount of time allocated to doing them was also very welcome and made the day feel far more productive than previous ones.
Xavier did a really great job of explaining mathematically challenging concepts clearly in limited time, and also in giving broader motivations for the field.
the exercises were awesome. They really helped me understand the material.
The day was very well structured and it was clear what we were doing when. The exercises were clear and helpful.
Both the presentation structure and lecturing by Xavier were great.

Would improve24

This day was really really good. The pace was fantastic- you could follow everything whilst being slightly challenged. The exercises were good too, and at the end of the session we were able to fully understand a paper (not just superficially) and be able to discuss its merits and limitations.
Maybe adding some animations and some proofs for the most important results in the appendix
Perhaps one example could have been presented inside the joint class. The lecture was clear! Perhaps it would be possible to compound the python exercises to try and reproduce the findings of https://arxiv.org/pdf/2405.15943 in a small transformer. More comments about open problem in the field would also have been useful to give more direction! Overall very good lecture and the topic fits well in the allocated time.
We spent a lot of time introducing and defining GHMMs, minimal sufficient stats etc. This material was very clear and pedagogical, but it might have been nice to spend more time talking about how this relates to learning, how it cashes out in LLMs, and what the important open questions are. I now know a lot about the data-generating process, but still have only a fuzzy understanding of why or when we should expect it to be embedded nicely in transformer activations and what implications this has for behavior and learnability.
I thought the teaching was very good, in fact I wanted more of it! Felt like there was a lot of time towards the end of the day for reading that could have been filled with an additional lecture / some other activity.
The exercises 1 and 2 were really easy. But maybe that's actually a good thing
- more time on the mixed state presentation, it went quite fast relative to how central it is - link the exercises more directly to the Shai et al. paper i.e show how the belief states from the HMM exercises end up in the residual stream
I think at points the material was quite abstract and I didn't really connect it to the feature analysis from say mech interp. Maybe also the applications to steering and how these tools can be applied felt fuzzy.
very nice lectures and exercises, maybe make more time for coding?
More time for exercises, as they were fun and useful.
Commenting more on the limitations and future goals
I would add more coding exercises and show resulting plots as outputs and explain concepts by interpreting them.
The lectures were overall clear. I think it would be beneficial to go through exercise and examples in the classroom.
Nothing really good day
a short explanation of IID, clear definition of hidden markov model at the start.
Hard to say honestly, such a great lecture, structure, the material and the day. Three things though: 1. Exercises -- maybe it's just me, but they required a lot of pretty simple compute, which becomes tedious at some point because you do not feel like you are learning or understanding smth new with the new exercise. Maybe makes sense to have a pointer like "if you do not have time to do the computations, take the resulting matrices from Solutions and proceed to the next steps." 2. Given that I have been catching (although maybe it's just me being unsure about the hidden state of mind of surrounding students) the "we can find the belief state geometry, so what?" vibe slightly, maybe makes sense to provide an empowering example, however small. The paper was incredible, don't get me wrong, but it does not look like it was found too promising of the research direction that necessitated the entire startup established under it. Smth to think about rather than just a suggestion, as that's the problem that was present in "physics of Deep Learning" as well -- people are like "So what?" forgetting that Quantum Mechanics and GTR was also pretty much "We know the deflection of light by Sun and additional 5% of precession of Mercury's orbit, so what?" and then we had precise GPS and gravitational waves. Nice to be showing that "So what?" is not a question to be asking. 3. it would be nice to make the intro lecture somewhat more connected to reality, as it feels quite abstract in the beginning. Say compare this with the hidden states of mind: we do not know the mood of the person, and we infer it from what they say. So we have this "mundane" problem familiar to pretty much everyone and then put the math behind it. Otherwise the beginning might read too abstract, and my brain gets lost when it does.
- Don't assume we've gone through the full exercise sheet when starting off the next lecture (he made a lot of references "as we saw in the exercises", but a lot of us didn't get this far, so it felt really out of context). - Spend 5-10 mins in the beginning reviewing expected knowledge (e.g. HMM for today, "beliefs", "belief system")
second part was a little too dense
I thought the material was very interesting, but I think I'm still missing a bit of information about how this could be applied / used for AI safety/alignment work.
I think this topic could do with two days instead of one.
Maybe a bit more on the foundational math.
- Adding more pictures, - introducing the definitions a bit gentler, I felt that other participants might not have enough relevant background to quickly grasp what's happening (i.e. physicist don't know Chomsky hierarchy)
The lectures we too fast. Make significant pauses before going from one slide to another. It gives time to take notes, digest the material and come up with questions about things that are misunderstood. The language in the exercises was hard to understand coming from a physics background.
I think perhaps the exercises. I found them very easy and often not very illuminating. I also struggled a bit to understand the applicability of computational mechanics to AI safety from the slides, but this became much clearer after a discussion with Xavier and reading the paper. Probably the day could have had more content - I think it is an interesting subfield!

Changed my mind14

I was not aware of computational mechanics and where it fits into the interpretability/alignment world - I would consider this as a potential area of research I would like to eventually contribute to.
I am more aware of the utility to study toy models to assess the learning capabilities of large models.
Somewhat more persuaded that this geometry of learned representations stuff is worth pursuing, although still seems unclear if it will extend to representations of the concepts we care about in the networks we care about.
Very interested to learn about belief state geometry and would be interested to think about theoretical guarantees in this area.
- interp on toy processes with known "ground truth" seems a lot more of an approachable problem than I thought, ended up picking this area for my proposal
I understand better the connection between computational mechanics and theory of world models.
Discovered a really interesting project at simplex.pub, and got interested on computational mechanics
I think computational mechanics is a very interesting field that I didn't know about until today. I will definitely dig deeper.
The reading felt a bit 'numerological' or like astrology but maybe it's just that I didn't understand it well enough. Will think about this.
I am sceptical about this being able to help AI Safety in the next 5 years but appreciate its value in scientific understanding for the future.
Hidden states wow. I knew that HMMs were used for financial markets in the second half of XX century and Renaissance Tech made a lot of money from that, and it's wonderful to see that they apply here too. I also realized that the states of mind reflect very well upon the material. We can be listening to the person / reading the text and be unsure whether they are passive aggressive or sarcastic or normal / whatever.
"pretrained models should learn more than the hidden structure of the data generating process—they must also learn how to update their beliefs about the hidden state of the world as they synchronize to it in context" that's crazy
It was interesting to learn about computational mechanics, although from the readings, it seems unclear how scalable it is.
I have been encouraged to think more about computational mechanics. When I first started thinking about understanding LLMs my first thoughts were a (very) naive version of this approach. I am considering looking into this area further and possibly pursuing research in this direction.

Other thoughts4

Thanks to Xavier for his time and cheerfulness!
n/a
I'm having some personal problems that were making it difficult to focus. But the lectures today seemed great - pace, difficulty, teaching, exercises, everything was really good.
N/A

World Models Fri 11 Sep, Fernando Rosas 39 responses · energy 3.7/5

Overall +0.30 better than expected n = 37

Content +0.05 as expected n = 37

Clarity 0.00 as expected n = 37

Motivation +0.05 as expected n = 37

Exercises +0.08 as expected n = 37

Readings +0.25 better than expected n = 12

Other activities +0.32 better than expected n = 19

Pace -0.19 about right n = 37

Difficulty -0.14 about right n = 37

Novelty +1.14 mostly new n = 37

Read what they wrote54

Worked well14

i enjoyed the group exercises
The last bit about cognition in cognition in LLMs. I hope we will have more focus on similar experiment driven research.
The fact that it built on top of yesterday's material was crucial to its value. But it was a very technical day that required more time to grasp.
The exercises let me understand exactly what was happening. The slides were gradually presenting information, making the lecture much easier to follow.
- The interdisciplinary angle i.e bringing in stat phys / neuroscience (a lot more intuitive for me_ - The Colab exercises!
Fernando did a great job at explaining very technical concepts and also in giving intuitions when faced with questions.
I think the slides for the lectures were well-designed.
I thought the introductory lecture was the best introductory lecture we have had so far. It motivated the framework in a clear way without assume too much background.
- The introductory lecture at the start of the day. Spending one hour to sent context before doing anything technical was great - Margot's talk at the end of the day is a great addition. Having someone talk about what they've worked on and the sort of work being directly applied to AI safety was nice.
Presenters were clear and concise
Mixing pen-and-paper exercises in steadily increasing difficulty with coding exercises.
It built upon the notions that we saw the day before!
The last part on metacognition seemed quite interesting (maybe not too mathematically though).
The real-world examples by Fernando, such as "Listening to a conversation from behind the wall" -- precisely what I asked for yesterday. They were helping putting the discussed material in context. Also the Colab exercise helped understand the topic and rebuild the math behind all this, very good idea. Also, the lecture by Margot was very satisfying as the day closer, where we went back to the high-level thinking.

Would improve26

for the group exercises we did some times we didn't spend time to discuss our solutions
Lectures were great. I suppose you can still feel whenever the lecture is taught by somebody with and without a substantial teaching experience. I'd like to spend a little bit more time to comprehend everything. As a meta-comment, it might be useful to split all the lectures in two days between two speakers. For example, the first part of the day is dedicated to the old material from the previous day, while the second part of the day covers the new lecture material. The idea is to allow people solve exercises a bit more, think of some questions and ask them the next day.
i'm not sure, but it would be cool to show/include more AI safety relevance; i felt that we basically just did the same math as yesterday in a slightly different setting and it felt a little bit pointless (<- that's a too strong word actually) to me?
There was quite a bit of overlap in concepts between yesterday and today.
As I suggested on Day 1, I firmly believe that a series of two-week "Iliad workshops" or schools on each of the topics covered in the Intensive would be extremely valuable as a whole.
Put the material online the day before. The exercises could be a little more wordy as I had to deduce a few things that could have just been written down.
- sharing the slides (incl. the handout version) and the Colab link before the lecture, instead of only via the QR code in the slides - the units felt a bit disconnected, a short recap tying them together at the end would help
It would be nice if the content and presentation of this day was more synced with the previous one. In many places, explicit comparisons could have been made as to how e.g. the formulas relate to those we saw yesterday, and I expect that this would have cleared up a lot of initial confusion and make the lecture easier to understand overall.
I enjoyed the content, the only change I would make is spending more time on modeern world models and why they work so well (e.g. what we know about "good" latent spaces for generalization and extrapolation etc), rather than the technicalities of epsilon-transducers - only given that the day was titled "world models".
I thought this was very similar to the module on computational mechanics, and it felt a bit repetitive without adding much insight. I thought the exercises were too easy.
There was some overlap with the previous lecture, I think this could be improved freeing up space for more material. The later part of the lecture was very interesting, as well as the introduction.
I wonder if this connects with other aspects of alignment, such as training dynamics or physics of deep learning. It connected with the topics of yesterday and was useful that they were built on top, but not sure what the advantages and things that the worldmodel approach miss are.
I would prefer if the teaching days were restructured. I personally find that having a single day on a topic is a bit too much of an intense burst. I would prefer if the day was cut into three parts, each with a different speaker. That way every speaker would be given the same time but it'd be spread out over 3 days. This would help with 1. Repetition: We would return to topics several times, helping with retention 2. Focus: Adding diversity to the day would help keep stuff interesting, especially towards the end of a long day.
Begin with a short summary of the previous day, and perhaps give more intuition for what we are trying to achieve
less today specific but about the entire week. we tend to have discussions every day about how the material of that day connects to ai safety and there seems to be a lot of confusion around it. like for example, there seems to be the somewhat implicit idea that finding the believe state in a transformer helps ai safety by understanding better how they represent data. but it would be very super cool to hear every lecturer explicitly explain why they think this is the most impactful way forward to make ai safer. currently it often seems a bit like research is blindly pursuing several directions that could maybe be relevant to ai safety. concretely I would wish for every lecturer to at the start motivate how this fits into ai safety and motivate their field from there, then present their field, and then at the end tie things back into ai safety and outline what they hope can be shown eventually and explain why this would help ai safety a lot. I wold also love for the lectures to explain why they have chosen to work on this particular field rather than other fields and how confident they are that this very field is the right one or if they're also somewhat unsure about which field could be impactful
I felt like we spent a lot of time today repeating the same material as yesterday, with slightly different notation and very different vocabulary. I didn't really see the point of introducing all this language for obvious-ish concepts until we got to the Factored Representations paper. So perhaps foreshadowing the relevance to LLMs and AI safety earlier in the day, and spending more time elaborating on this material, would've been better for me. Perhaps a few examples illustrating whichever definitions were necessary would've been good too. But I also had a headache all day, so possibly my outputs on this feedback form are not causal consequences of my perceptions or predictions. Please take this feedback with appropriate salt.
Just some notational inconsistencies. It was a great day.
Add corrections to the exercises in appendix
I think some diagrams were difficult to understand. I would have enjoyed discussing J-lens in more detail (also focusing on the actual maths behind J-lens) - I think the maths is kind of unprincipled and makes me question the validity of the claims made in Anthropic's paper. I would have loved discussing it at Iliad.
I thought it was a bit of a mumble jumble. We covered generic "world models", then went to computational-mechanic flavoured stuff and finally did a bit of a lit review on metacognition work. I don't think this worked out. The content has to be better designed and organised.
the lecture felt scattershot, unfocused and lacked rigorous definitions (which IMO are essential to understand the concepts covered today, by unpacking them and giving concrete examples). There were several errors in the slides that hindered understanding. The connection with yesterday's materials (computational mechanics by Xavier) was made but not as well as it should IMO. Also I don't see why world models were necessary for the lecture.
Last 2 presentations were too fast and I would have preferred a more explicit link between the theory and the applications (last part of the day)
Some more explanation on how can human contribute to these research topics competing with LLM. Why we still need human for these tasks, and in what sub-topic would human-participation would be especially valuable.
Slides numbering. Also, I think we should normalize the "Ohmmmm..." introduced by Julian so that we can finish breaks with it more smoothly. MSP was not defined before giving an exercise. Exercises were way too abstract in the beginning, but did improve significantly quite soon. The material itself resembled yesterday but felt much more abstract conceptually. The intro lecture had all this notorious RL formulae that are hard to interpret on the fly, especially without examples that map what formulae convey with each symbol. Part II with transducer for me was hard to follow because despite great examples given by Fernando throughout it felt very abstract and disconnected from reality still. It might well be that my brain is just not made for this kind of abstraction specifically though. Part III was much more computational and accessible to me, but some concepts (MSP) did not look too well defined to wrap my head around them. Finally, the produced fractals, as a result, did not click. Yesterday is a good comparison -- I was in awe. Today, however, I was utterly lost on what makes this promising.
nothing
Very minor thing: would be helpful to make the terminology between yesterday on computational mechanics and today consistent (belief states vs MSP; i don't thin they were explicitly called \epsilon-machines yesterday?) Otherwise, I thought today's lecture worked really well with yesterday's lecture.

Changed my mind11

world models are the next big thing!
For a long time I've seen work on agents as either theoretical that narrows to tabular RL (which is kinda boring) or purely empirical. The lecture and connections to computational mechanics as well as discussions on world models helped me reassess some of these views and think of new connections, especially to tensor computations I've been working on for a while.
I updated my definition of a world model.
The J-Lens!
I found the world model material very interesting and I plan to look into this material further.
n/a
it seems there is some real depth in this research agenda of computational mechanics. compared to mech interp I like the idea of building up synthetic data as ground truth and studying whether an LLM represents this data in the optimal way which it seems to do. I am still somewhat confused about a lot of the concepts covered today but it feels worthwhile digging deeper
I am more confident that this field has impact on AI-alignment and safety. While finding it fascinating, I was afraid it was too theoretical to be useful in the short term.
It motivated me to check some frontier papers. Also discovered some AI programs throughsome other participants.
This side of comp mech did not convince me to be worth pursuing, most likely because, as Fernando mentioned, it's hard for the bottom (Transducers) and the top (Metacognition / personas) to meet in the middle, at least yet.
I believe this question should be written 'Did you change your mind about anything today?' or 'How did your mind change with regards to a certain topic' or something. Anyway no idea still figuring it out

Other thoughts3

n/a
Today's schedule is a bit tight
To make it more general: I think we should normalize the "Ohmmmm..." introduced by Julian so that we can finish breaks with it more smoothly. As otherwise lecturers need to be more loud to call people in, and that is not as effective

Decision Theory → RL Mon 14 Sep, Fernando Rosas 38 responses · energy 3.6/5

Overall +0.14 as expected n = 36

Content +0.28 better than expected n = 36

Clarity +0.08 as expected n = 36

Motivation +0.14 as expected n = 36

Exercises -0.25 below expected n = 36

Readings 0.00 as expected n = 13

Other activities -0.11 as expected n = 9

Pace -0.17 about right n = 36

Difficulty -0.22 about right n = 36

Novelty +0.67 mostly new n = 36

Read what they wrote50

Worked well17

- Grounding RL in preferences instead of just starting from rewards
The exercises with solutions. The formal introduction of the axioms with examples.
Less coding exercises: I think that this was a good choice given the material.
I liked the structure of today's lectures. Starting from the fewest assumptions possible and building up to something that has enough structure to prove results is very intuitive.
Exercises like Problem 2 where we actually did compute some stuff, plus a few other like Problem 1 which taught us how to derive the Bellmann equation -- helps invaluably.
I really appreciated the first lecture. I have read intro books on RL before but never delved into how the structure of preferences determines the reward function. Very interesting!
exercises! very fun to do some math
I really enjoyed the exercises.
We had all the materials sent beforehand
Lots of exercise time, Leo was very helpful in particular.
vibe-reading by my own the materials
I think the slides were structured well and there was a natural logical progression in the results being taught.
i liked the agency part surprisingly much
The logical structure of how stuff was presented made sense, we built well from one idea to another.
The initial discussion about what agency means, and how we should model it, and the introduction of the von Neumann framework.
not sure, spent the whole day being pretty confused about preference and utility
I thought the the connection between the fundamentals to the broad strokes of RL was illustrated with a lot of clarity.

Would improve22

I felt like introducing such practically relevant content in terms of axioms was a bit terse. I would've enjoyed it more if a more practical approach was taken to the content
More concrete examples
Good class today, but it felt too self driven. The lectures were relatively short (2.5 hrs) in total maybe, and the rest of the time was dedicated to independent exercises. I think it would have been more useful to maybe cover some relevant problems or applications as part of the lecture.
- Would have liked a paper or textbook chapter to go with the lecture (e.g. Sutton & Barto ch. 3 for the MDP part) to have something more formal to work from - More worked examples connecting the VNM axioms to reward / return / policy
The day was quite abstract and very proof-based rather than compute-based. I personally was quickly losing interest in the exercises as I did not see value in doing those proofs. Perhaps would be worth considering to have problem sets for differently inclined people: smth like Problem 2 but longer and much harder for applied mathematicians / theoretical physicists, some Colab notebooks with simulations / visualizations for more computationally-inclined students, and indeed proofs for pure mathematicians. Finally, just a selection of conceptual readings like LW posts or arxiv preprints to see what are those non-Euclidean preference axioms in practice for more philosophically inclined ones. Also, what I lacked was a more empowering example. Maybe it is hard or even intractable to provide one, but say showing how it works in say AlphaGo or another familiar system would be helpful to see the value in this. Also, the notation in problem sets: what is that infinity under the || metric?
The day was too sparse, too much of unstructured activities
This lecture was much better than last Friday's. It covered concrete set ups and I felt like I had some sort of closure after each of them. However, this feels quite disconnected from how people do RL, ie. it would've been useful to do some coding (we have covered the Arena material essentially so maybe presenting the corresponding coding exercises as alternatives to the theoretical exercises would be useful). Also, I am not too sure how RL is done in practice, ie. how does deep-RL work? How do we optimise policies in these cases, etc.
Add a third lecture - more material!
Liked the introduction to rewards in the first part, I was wondering if it can be compared to other approaches and discuss a bit more where it fails/what it can't capture.
maybe some discussion about the exercises as a group so we keep the energy more focused, but the exercises + lectures mix is personally so much more effective than lectures alone
I particularly enjoyed the second part. I kind of see the motivation behind the first part, but felt the assumptions where quite unrealistic, the notes were great, though.
I would have liked a few examples of contemporary research questions around the topics we covered today
I especially liked the first part related to introduction to utility functions and derivation of the common reward optimization framework from the common assumptions. This part was definitely valuable. Overall, I tend to believe that the core part of learning and teaching given that we all have LLMs right now is an ability to identify corner cases, limitations and point people to them. On the other hand, it might be useful to dedicate at least some time to Deep RL and some ideas and limitations that break in when we start dealing with finite datasets, off-policy learning and functional approximation instead of purely tabular settings.
Too much time for the exercises. I would have liked more explanation and discussion time.
I think maybe a few more slides on Q-learning, as it was a little more difficult but seems like quite an important aspect of this theory. I thought the exercises where quite easy apart from Ex. 6 which was quite tough.
i would have liked solutions for the exercises or discussion for what we should take away or should have taken away (afterwards)
I liked having the schedule for the day on previous days. One could also consider splitting up the lecture time more, since the exercises ran very closely along the lecture, so alternating more frequently good be helpful, as it would ensure that previously established concepts have been internalised before moving on. As it was, the wait between "you will do this in detail in the exercises" and the actual exercise was a bit long for later topics.
I found today's material pretty interesting; I've seen a lot of it before in some form, but this was a clearer and more complete presentation. However, there might be some room for improvement in pedagogicalness. To some extent it suffered from the classic "math's pedagogical curse": Equations and proofs were clear, but it took some work to read the underlying concepts/motivations out of the equations. For example, maybe some schematic discussion of MCTS would've helped give a clearer intuition for the Bellman equation? And certainly vNM/Dutch Books lend themselves to cute examples about drunkards and gamblers with irrational preferences. Generally any excuse to put a diagram on a slide full of equations sounds good to me.
- Maybe swap one of the analytical exercise sessions for a hands-on implementation of RL in a simple setting (e.g. where the set of states is small) - I would have liked to see more in detail when RL in the von Neumann framework fails, and what are the open problems
The language and notation in the exercises are difficult to understant. It's a pretty mathematical day as well, I'm not sure I saw the direct link to AI safety.
I would try to have more practical examples for the material. Plots trial systems, anything more practical to illustrate the math.
I was already familiar with most of the material and so I decided to read some papers from Simplex during the lectures/exercises.

Changed my mind9

I appreciated in particular the pre-lecture reading on decoupling beliefs/desires. That persuaded me that our theory of agents loses little generality for the common-sense meaning of agency.
That the entire field might be as misguided as Economics due to expected utility theory. The ASI most likely won't have consistent preferences because you don't need them, just like humans survive easily without them. No one Dutch books anyone or money pumps anyone because the effort to discover the inconsistent loop requires NP-hard computations, which no one ends up doing. Thus undermining the motivation behind the field and most justifications.
n/a
really curious now to learn how Q-learning was applied for the bot that plays starcraft
I discovered that even though RL is not the main technique frontier models rely on, it might be helpful to understand it because it is used in post training.
agency is much weirder than I realized, i find the vNM axioms much less obvious than before
RL really doesn't tell us much about alignment per se!
Did not get time to reflect on anything
RL can be worse than supervised learning in a lot of cases.

Other thoughts2

Solomonoff Induction + AIXI Tue 15 Sep, David Quarel 36 responses · energy 3.6/5

Overall -0.09 as expected n = 35

Content -0.23 as expected n = 35

Clarity -0.23 as expected n = 35

Motivation -0.43 below expected n = 35

Exercises +0.09 as expected n = 35

Readings -0.13 as expected n = 16

Other activities +0.14 as expected n = 14

Pace +0.40 too fast n = 35

Difficulty +0.26 too hard n = 35

Novelty +1.40 mostly new n = 35

Read what they wrote54

Worked well18

I think both the lecturing and the notes were very good.
exercises with a partner really made a difference!
It was easy to ask questions and the answers were helpful, but I don't think that gathering almost all the interesting information in personal communication in such group sessions it good
The lecturer was clear
- David was a really energetic speaker, made a heavy topic easy to stay engaged with - Good worksheets, and the balance of short lectures with lots of time on exercises
I like that it used policy and Bellman equation -- the material from yesterday. Same goes for Thursday-Friday transition with HMMs, and Friday-Monday flown to Tuesday seamlessly. Very nice. Also the nice and long list of exercises to work on by yourself.
I liked that the teacher was very eager to answer questions, I liked the explanation on KL divergence, and him recommending me a concrete section in the book that exactly answered my question
David is clearly excited about and deeply interested in the material; he did a very good job with questions both during and after presentations.
The website content is FANTASTIC, well done.
Personalized explanations.
Long exercise sessions dedicated to learning by doing. We went over all proofs ourselves rather than watching someone zoom through a derivation. Was very nice.
The clear layout of notes and encouragement to work in a way that suited. I also enjoyed listening to the questions asked by other colleagues.
working through the exercises which seemed very well written out
The abundant time for exercises and discussions.
The written lecture and the exercises were great, well written, and built up to interesting result. However, I question the usefulness of relying entirely on exercises to carry the core information. I am not necessarily interested in solving mathematical puzzles, but see how the experts in the field do things (because one day I might collaborate with them). For example, several exercises were either straightforward or tedious according to notational rigour, and I would have preferred seeing the lecturer carry out at least some solutions or proof by hand to see what is the style of the subject.
The website to the lecture materials and the problems was very accessible and the sections were well written.
The exercises were fun but still for someone with a background in bayesian stuff it wasn't very new, but perhaps for others it was useful.
The focus on the exercises + the incentive to work in pairs

Would improve24

Maybe a more balanced distribution of lectures and exercises - while I found the exercises to be interesting and reasonable in difficulty, it was a bit "intellectually exhausting" to work through them for multiple hours. I also would have liked to see more material on the current research paradigm in AIXI and what it hopes to achieve.
Needs a bit more context and applications. It all seems a bit far from what is happening now in the world.
It's surely teaching style, but extremely short lectures, not engaging constructively with questions, and exercise sessions that are mostly symbol juggling together don't show how this all is relevant to safety and alignment.
I think adding more lecture material and not deferring everything to questions and exercises could be much more helpful. If felt like we were given some abstract nonsense introductions and definitions, and had to clarify everything ourselves. The connection to alignment and reasonable agents we observe and want to make predictions about it also kinda vague. If there are known results about AIXI application to known environments (e.g. Markovian), adding them to presentations and allowing to play with them locally could be much more convincing.
I really enjoyed the lectures. I think today would have benefited from more lecture time. The exercises, although nice, were a bit difficult to parse (there was a lot of notation) and probably something that would be better to look at in our own time.
- A bit more intuition before the proofs i.e what the Bayesian mixture is doing on a small example
This day's structure (i.e. very short talk in the morning and afternoon and then exercises) did not work the best for me. The exercises were mostly around proving the results from the talk but most of the times I didn't know if I was proving something that was just a middle step for the big proof a few exercises ahead OR an important result that I should be able to interpret in the big picture of Solomonoff Induction/AIXI. I think a few changes that would help me would be (a bit) longer more comprehensive talk AND/OR more discussion in the notes in-between the exercises. (Please note that I'm quite in favour of the philosophy behind this structure, I just didn't feel it was accomplished in the best of manners)
I feel the model is too far from any computational process that could be implemented in the real world to be any useful
Today was a big jump in terms of usage of technical terms. Complexity, computability, etc. had not been discussed previously and this is a hurdle for those with a weaker background in computer science, including the quiz at the end of the day, likewise (but to a lesser extent) during the lecture. Regarding the lecture, I strongly believe that taking more time and perhaps showing one of the proofs (possibly partially) would help. This could also break up the long exercise session during the afternoon. But generally slowing down the pace of the lecture would be good. I heard from multiple people that they had to go through the lecture material again, because it was too fast in the morning. In my opinion, it is important to answer questions and confusions during the lecture even if that means slightly adjusting the schedule.
sadly for me the lectures were much too fast, felt quite very rushed, there was little space to think about what's on the slides and I left both lectures not knowing what actually just happened :(( I would have benefited from a less rushed presentation, more motivation of what we are doing, more coming back to what all the notation means, and importantly again how this fits into ai safety research. we asked how the content today is relevant to ai safety and David provided a reasonable argument but I really think for such highly theoretical days it's super relevant to embed things into an alignment perspective
The presentation was generally quite good. However, it introduced several ideas at once; I wonder if it would've been clearer to separate them. We could've started by discussing optimal prediction and optimal behavior under an arbitrary prior/hypothesis class. Most of the results we discussed can be two-thirds proven in that setting anyways. The Kolmogorov prior is cool, but I think it makes more sense to introduce it afterwards, since it comes with a bunch of philosophical and technical questions. I felt like these got somewhat mixed up with the relatively straightforward general-optimal-agent story. Also, more discussion of how this relates to AI safety would be good. I'm not sure if this is a complaint I have about today's presentation or about the actual state of the AIXI field; quite possibly the latter. The exercises were reasonable, but not a great notational complexity to conceptual complexity ratio. I ended up spending a lot of the allocated time working on unrelated stuff anyways, so maybe the later exercises were better.
This was a really interesting day. There is a lot of material so I would go back to two days (delete the world models day!) and this would allow you to cover too recent papers. ie. https://djsutherland.ml/papers/nfl-ait.pdf seems quite interesting and show a bit what AIXI labs is doing. From the lecturer, we got the impression that AIXI is "old stuff" that might not be useful, but seeing what these people are working on might give another viewpoint. Overall the format was good, I enjoyed the zap lecture and then doing a lot of self-study. Maybe going together through a paper (e.g. the one I suggested) elicits a deeper understanding of the material + fruitful discussion between people.
Too much independence, and the lectures were so fast. I would have liked more time for teaching and less independence.
After a week of mainly mathsy work I'm really yearning for a link to practical alignment. How could these frameworks apply to analysing transformers? How are these methodologies applied for interpretability? How would the interplay of theory and empirical work look?
a bit rushed at times and sometimes I wanted to hear the question and answer as I would not get to hear it later.
more discussion about the answers as a group :)
A better way for slides to be presented. The TV screen is simply way too small and not conducive at all for a presentation to 40 people.
I would like longer sessions and less time for exercises. Overall I find this approach a bit too formal, but this is personal taste
The room layout and lecture setting was more conducive to coworking than teaching. This was clearly by design, but I found the format offputting for a completely new subject.
I would show more concrete empirical research examples instead of just going through the theory.
In my opinion, today was too much exercise-based. I would prefer to discuss the theory in detail in class, so I can do exercises in my own time later. The first few exercises were just math manipulation which I personally find a bit tedious and not very insightful. I would much rather learn the theory in the context of AI safety, rather than proving theorems.
There were many exercises that were not giving the insight I would have expected and the notation was a bit too heavy. There were to some extend a bit of overlap with yesterday's session, but the theory was still interesting.
Talk was slightly too fast for me but I understand the philosophy of the lecturer. But maybe a tick slower, but then again I was slightly late so its on me.
Maybe adding one or two recent paper to showcase current research in that field

Changed my mind10

I think AIXI seems fairly promising, but I didn't feel like I got much context on what exactly it aims to prove.
not sure
n/a
That even so called weak assumptions might not be that weak. Also, infinite compute, really? I do share the sentiment that we have to be studying idealized agents, but significant amount of time should be devoted to realistic scenario too. And that's where the math has to be employed the most, as all these theorems might well be inapplicable (like no one in the market cares about the existence of Nash equilibrium because no one bothers computing it and you can't really arrive at it in reasonable time).
How useful aixi is
I used to think SI smuggled a very strong assumption into the choice of the Kolmogorov prior. Now I think it smuggles a very strong assumption into the choice of UTM instead. (This is a real improvement in my eyes.)
This feels particularly useful to derive mathematical bounds on agent behaviour- but not a lot of people work on this although it seems quite interesting. I feel like this can be more promising than things like computational mechanics (which feels like fancy mech interp). I think the roadmap on how to use AIXI is somehow fuzzier but more promising.
I discovered AI safety initiatives in Barcelona, my city.
This work is too theoretical so having an impact.
I understand now why AIXIs might be useful to AI alignment, albeit not immediately.

Other thoughts2

n/a
Today I spent most of the day reading Simplex papers because I found that much more interesting than the Solomonoff stuff.

Training Dynamics Wed 16 Sep, Guillaume Corlouer 37 responses · energy 3.6/5

Overall +0.09 as expected n = 33

Content +0.30 better than expected n = 33

Clarity -0.06 as expected n = 33

Motivation +0.18 as expected n = 33

Exercises +0.12 as expected n = 33

Readings +0.06 as expected n = 32

Other activities -0.04 as expected n = 24

Pace 0.00 about right n = 33

Difficulty 0.00 about right n = 33

Novelty +0.91 mostly new n = 33

Read what they wrote47

Worked well14

The exercises were good
Really enjoyed the content on learning dynamics today, was mostly a review for me but was well taught and found the exercises a very useful review.
paper reading + discussion with people after is always sick, we should have this every day imho
working out the examples and discussing the paper with the other intensivists :)
Deep linear networks intuition for lazy vs rich regimes.
The day being split into a lecture half and an exercises half was really nice.
The paper activity was great. I like when we have exposure to papers and read them properly. I also like how some days continue into others, like today was a great supplement for Physics of Deep Learning on Tuesday, and a nice continuation at that. Also, liked mentioning Noether Theorem and the approach to the problem via symmetries -- very physics-friendly. Allows to look at the problem from different angles and using different approaches, which is mathematically invaluable.
I liked that we had a good mix of theory, exercises and empirical analysis of the applications. I think this day had the best split between these three of any so far.
The topic is very interesting. But the materials provided to absorb the content and learn was not. It would be good to rethink this!
I liked the edge of sharpness/learning rate limit result.
exercises reading person seating on the LHS who explained some things more clearly than the lecturer
The reading lists were useful, but I believe it was overly ambitious to reach the stage of informed group discussion after devoting so little time to actually reading a paper.
Lecturer very clear.
Reading by myself.

Would improve21

Exercises were surprisingly easy. Maybe remove hints? I think splitting the lecture in two would be better for maintaining the overall attention of everyone. Some parts of the lecture were suddenly very mathematical and some of the important notions weren't reviewed. It would be good to give a bit more context on Lipschitz stuff?
More time on NTK derivation steps.
Split: theory morning - exercise afternoon is not optimal in my opinion, I did enjoy the exercises and coding a linear network was also useful to see the theory in practice.
Have more that one speaker on a given day. I think I'd prefer having two/three topics per day with each topic's content spread over two/three days.
I think two lectures in the morning one after another was quite intense. The material was good and the delivery too, but it needs slower pacing to sink in. As I caught myself thinking on one of the last slides "wow that's a lot, can we please stop here," as each slide needed some time to be digested. Yesterday was an exemplary structure: lecture, then exercises, then lunch, then lecture and exercises again. Today would've benefitted from it too. I would make exercises slightly, just slightly harder. Perhaps there was no need to make the matrices diagonal, or maybe there was. Also, on paper activity -- maybe having a small workshop (30-90 mins) on paper / LW post reading (first (and last) sentence of each paragraph, then in full) would be beneficial. How to read with the best retention / understanding given time constraint and objective.
I found the lectures were quite difficult to follow due to the density of technical material, but it did nevertheless help clarify the reading later in the day. I don't think the content itself was necessarily too difficult, but I struggled to grasp it from the lectures at least. Perhaps due to too many interruptive questions from the audience. With the reappearance of the NTK, I think in hindsight it would have helped for it to be a larger focus during the Physics of Deep Learning day. I discussed it a little with others, and I didn't feel convinced any of us had understood it deeply.
- the visual (slides) and verbal (speech) parts of the talk were a bit too disparate, which made it difficult to follow - it would be nice to have a subset of exercises to prioritise (on top of this, the fact that before starting the exercises I wanted to review and understand the slides took me some time) - like yesterday, I feel like last week's average day structure was preferable to the way today's structure was implemented (in particular, I prefer more talk time intercalated with other activities) PS: I think that less time for exercises makes the "lecturer" think them more carefully, rather than assigning an unattainable amount of exercises
I would split the lectures in the afternoon and morning and maybe talk a bit more about the stochastic part
I'd like to cover more material during the day. Going over differentiation and notions of gradient and hessian over the day on a course for math and physics PhD graduates seems questionable -- it's like a one undergrad university lecture in a day. Going over some more experimental exercises could be great as well
Using Claude to generate slides is fine, but these have to be checked. Figures are only sometimes cited (in vary different forms), making it difficult to go to the original figures... Some figures I couldn't even find after looking at all the references in the references slide! Also, the exercises were "fun" but they weren't super useful I think. It was valuable that they showed us the maths of the results we saw during the lecture. However, there was too much focus on this which meant that we didn't have enough time to read and understand the papers. Change this!
I think it was a bit optimistic to read some of the papers in the given time. Maybe a guideline to each paper would have been helpful.
I thought the day was quite good. Maybe add a more practical exercise.
Maybe slightly too much discussion during the lecture. I would have preferred to cover the essential ideas first before going into "what if's". As it was, I lost track of the bigger picture and the current train of thought within the lecture material multiple times. The fact that I was tired certainly didn't help. I also think that it would have been great to have this sooner after the "physics" day, since many of the ideas from that day were picked up here.
General feedback: Some people today kept asking questions that were tangential, persistent or not particularly good and this disrupted the flow of the lecture and made the experience overall worse for most people in my opinion. Given the material covered each day is a lot, I think it's important to stick closer to the schedule and lecturers can answer such questions after the lecture; maybe the participants should be explicitly encouraged/reminded to keep some questions for after the lecture?
maybe adding a small project for people to train and compare their own deep linear networks.
there was not enough time to read the paper - the discussions were not very productive because everyone was just lost and confused
I'd like to see some more fine-gradined look at some papers, examples of training dynamics and maybe a few notebooks to reproduce experiments and play with toy tasks without reproducing them with Claude.
Unfortunately, the main lecture venue is impacting my ability to enjoy the taught sessions. The screen is small, we are crammed with no desk space, and it is hard to hear the lecturer. Any improvement on that would be greatly appreciated. Regarding the lectures, Guillaume did a good job at structuring the material, It took me a while to get the hang of notational conventions, but it was fine in the end.
I thought today was very good. I would have liked a lecture in the afternoon.
Too long lecture sessions without a break. I was not able to follow. Also the time for the reading was too short, we ended up discussing only section 1 of the paper.
The talk included a neat survey of a bunch of results that are out there in the literature, plus a more detailed discussion of some properties of deep linear networks. The exercises + reading gave a pretty complete and concrete understanding of the basic results about the latter. It was a little hard for me to understand the significance of the several papers that were mentioned or guess which I might want to read. I suppose I'd also like to have a little understanding of what the big open questions/short-term goals of training dynamics research look like.

Changed my mind9

Grokking understood as phase transition rather than mystery.
I'm more convinced today about training dynamics's potential as a helpful tool for AI safety. That said, I'm still a bit unconvinced that the mathematical frameworks introduced thus far are helpful for alignment. I'm yet to see how they can interface with empirical work in a way that allows theory to be developed in a way that compliments experiment.
The usual -- Learning Dynamics might be lost in math at times. As a theoretical physicist, I appreciate the bottom-up approach of going from simplest to harders, but as a theoretical physicist I know that it tends to lead into focusing on the simplest too much due to math being tractable.
I think what we covered today was very interesting, and I can see how it could help with understanding neural networks.
I'm interested on what was mentioned about tensor networks. I will definetly take a look into it.
Misalignment could be a more general phenomenon than I expected previously.
Given my previous background in optimization theory, I struggled to find the most promising application of it to AI safety research. Training dynamics seems to be the most promising right now!
n/a
I learned about Swiss AI safety days, maybe worth applying.

Other thoughts3

We need a projector for lectures
Great module.
n/a

Singular Learning Theory Thu 17 Sep, Kai Ogden 37 responses · energy 4.0/5

Overall -0.06 as expected n = 36

Content +0.17 as expected n = 36

Clarity +0.39 better than expected n = 36

Motivation 0.00 as expected n = 36

Exercises +0.28 better than expected n = 36

Readings -0.38 below expected n = 8

Other activities -0.18 as expected n = 11

Pace -0.83 too slow n = 36

Difficulty -0.78 too easy n = 36

Novelty +1.08 mostly new n = 36

Read what they wrote58

Worked well19

Excellent pedagogy and big picture motivation.
I liked that it was on a whiteboard, it made it much easier to follow. The teachers could explain things really understandably. I liked that they worked through some toy examples on the board. I also really respected their epistemic humility and it helped create a great atmosphere for asking questions, admitting confusion etc.
I think the short lecture followed by exercises system was quite good, but everything was a bit too slow - could have done several of these blocks probably rather than just three.
I liked the pace and especially the whiteboard lecturing. I follow better when new concepts are thought using a whiteboard vs slides.
The exercises well well chosen.
I really liked the bite-sized sections of lecture+exercises!
the long discussion at the start of today about how the theory today fits within ai safety, what the hopes are how it could help etc before even going into the theory details. every lecturer should do this!!
The whiteboard -- I did like the format a lot, reminded me of Uni lectures and the "How to Speak" lecture by Patrick Winston https://ocw.mit.edu/courses/res-tll-005-how-to-speak-january-iap-2018/ where he claims that comprehension speed is about the pace of writing on the board. Previous lectures weren't as easy to follow because of slides, while here you can write along and digest more slowly. I also liked that the motivation for devoting so much time to singularities specifically was given. The lecturers today had a vision that most of the time should be spent on singularities and implemented it. The exercises were also quite fun, I enjoyed them a lot. As well as the format, aka lecture-exercise-lecture-exercise.
The willingness of the lecturers to help during the exercises to make sure nobody would fall behind
i kind of liked the blackboard lecture style a lot (compared to a presentation with slides)
Recap of exercise 3.4 on the board was very useful to get insight.
my favorite day so far, the flow between teaching, exercises is quite good with the teaching sessions not too long and well designed exercises to gain some insights to the material hands on.
Discussion of the solution on whiteboard
I think the lecturers were good and engaged well.
Geometric intuition for local learning coefficient.
Short lecture, dive into exercises
Discussing with the lecturers in the off time clarified most of my questions , they were very helpful!
Kai is fantastic. He's an excellent communicator and clearly has some mastery of the topic. Matthew as well was very easy to ask questions.
Whiteboard is the best. Makes the pace perfect.

Would improve27

I think the class was pitched at the wrong level. Too much time was spent on trivial things like curve sketching and computing derivatives. On the whole, I feel that very few concepts were covered today. This was very pedagogical and I was able to follow everything. However, the level of this class was more appropriate to an undergraduate class rather than a class of PhDs and postdocs.
The intro & motivation was a bit too long. I think the pace could have been a bit faster, and this would have allowed us to cover more material.
I think the first two lectures were slow in terms of delivery of new concepts, and I think it would have made sense to have slides so that walking through the examples more seamless than it ended up being. The content was interesting overall and some of the exercises were good.
Some of the explanations at the whiteboard were not very clear and some others were a bit too elemental. At some points I think that the questions of the participants were not really relevant to the explanation, so I think the speaker should not be afraid to delay the answer and continue with what he wants to explain.
The initial discussion was great. However, the lecturer wasn't comfortable with a lot of the maths involved- which is in principle okay (maybe less okay in something like SLT though). The problem was that people kept asking irrelevant technical questions about some of the constructions. A lecturer with more mathematical maturity maybe would have cut off these discussions (they were mostly useless and pedantic) and would've continued with the intuitive explanations he was giving. Despite this, the materials were really good!
I think the day wasn't dense enough in material -- most of the time we just practiced differentiation and multiplying jacobians by fixed vectors. It's fun, but not for too long. Increasing the density of material and covering certain corner cases, where things break overall, could be valuable no matter what the second day covers. Adding examples of models from standard statistical courses and discussing where we could break them for SLT and what does it allow us to analyze could be interesting. E.g. showing some key non-identifiability results + LLC for these models as a comparison. If for them, unlike neural networks we discuss later in the course, things start to break, that justifies an additional discussion.
I think the pace was rather slow and would have liked to see more of SLT or more technicalities. We looked at examples of losses that were fine theoretically, but would have liked to see some of this formalism applied to real networks, even though just with approximation.
I was happy with how today went.
Lectures only covered a general overview, and hinged on too simple cases. The exercises were useful, but it would be useful to see some deeper and technical results during lectures as well.
The discussion at the start of the day was nice, I think having more time to discuss these broader questions would be nice, not necessarily on this day. There seemed to have been a misunderstanding in what the point of the lectures was, as I believe they were meant to provide a schematic overview with exercises going into the more specific details. Many participants seem to have expected more formal details during the lecture, and clarifying this disctinction at the start would probably have made everyone accept some level of glossing over details more than they did. This time could for example be spent on going into more detail on the particularly complicated stuff, like the volume scaling law. I think the content of the first block about singularities could've been condensed more, as most people are familiar with the underlying maths.
Add the usage of https://www.geogebra.org/calculator or similar (say contour plots) when we want to discuss surfaces and landscapes for the ease of visualization of valleys etc. The day started with the oral motivation for SLT (no math), and I found it not too easy to follow given that there were no slides. I think that part would benefit from slides. Also, I hope it's gonna be addressed on Day 2, but if not, I'd say that an "empowering example" or a success of SLT to be shown would be a nice point to start. Actually, applies for every topic we cover at the intensive. The lecture would've benefitted from more preparation of the whiteboard lecturing itself, as Matthew was sometimes unsure of what he was going to show, or at least that was an impression. Also, I am against any lecturing right before lunch -- too intense for hungry brains. The whiteboard would've benefitted from elevation, as writing on the lower part was not visible to the participants in the back. Also hope it would be addressed tomorrow, but showing how specifically statistical learning theory fails that we need SLT would be very helpful and motivating.
Having short slides on top of the drawings on the board during the oral presentations to help the audience following
- I felt like the talks often lacked structure, which contributed for the unnecessarily slow pace; also, more content would be nice - some exercises were too much undergraduate Calculus-like and the "insight density that they offered" was small - also, I appreciate the presentation + exercise structure of the notes but feel like given the small extent of the presentation part, it would help to clearly separate which results I should retain from something I learn from an exercise which has a particular context; (also maybe some plots would be a nice addition)
Generally was good. However, SLT seems to be a combination of (1) conceptual insights into loss landscapes and learning dynamics and (2) rather technical mathematical results and conditions that are useful for understanding, deriving, and evaluating the conceptual insights, but aren't necessarily otherwise important. It might've been nice for the teaching to distinguish more clearly between these. Which parts are technical waypoints and which are the big ideas?
I had quite low expectations, but overall the material was quite clear. It is, unfortunately, mostly undergrad calculus and algebra, hopefully the next day will give better perspective on things.
Most exercises were warm ups, very quick to finish and far from realistic settings. The lectures were not as substansive as I would have hoped. A lot of discussion that was tangent to the goal of the day. The lectures felt a bit unprepared. Definitely use varied pedagogical supports (slides + white board) to improve teaching. Too much hand waving without visual support to structure and ground the arguments
honestly today
Visual aids, slide wth the visualisations of the graph
I think some of the material could be better motivated, and also some pre-reqs could be assumed
Pacing on resolution of singularities.
The beginning felt too easy and the first exercises also easy. I would have liked more motivation and more difficulty.
I would make the material a bit higher level. Possibly also the connection to statistical physics could be highlighted more. Motivation/introduction went a bit overtime.
First half of the day with Mathew was fine but it didn't seem as prepared.
We should have more coding exercises.
Today was a really nice day. I loved that it was at the whiteboard, was great for setting the pace.
I think some of the mathematical formalism could have been made a bit more rigorous for clarity, although I think the overall ideas were intuitively conveyed. The pace was also a bit too slow and wecovered relatively little content vs. most of the time on exercises.
I found the pace too slow and too easily disrailed by questions that were not really important. Overall it was set at a level not appropriate for the course. The material was in principle new, but it over emphasied low level concepts and progressed very slowly for someone with 'mathematical maturity'. Some of the digressions were brought about by participants

Changed my mind9

SLT looks like the String Theory -- lots of math, some captivating some complicated, but with no practical utility. I have asked Matthew whether SLT made one apriori prediction and if it was successful and he told me he knows none. He did say though that SLT helped him see the things differently and in a richer manner -- more emphasis on that should've been made.
SLT is interesting, i need to learn more calculus
Nothing, we did not get very far into SLT.
SLT is very cool and i want to know more about it
n/a
Degeneracies are features, not bugs, in learning.
Perhaps SLT may be useful
I discussed with other participants about EA Global New York. I learned how algebraic geometry can be useful for AI safety.
nop

Other thoughts3

The schedule format in the google doc works well.
n/a
Excited for day 2.

Singular Learning Theory Fri 18 Sep, Kai Ogden 37 responses · energy 3.4/5

Overall 0.00 as expected n = 30

Content +0.07 as expected n = 30

Clarity +0.07 as expected n = 30

Motivation -0.03 as expected n = 30

Exercises +0.13 as expected n = 30

Readings +0.11 as expected n = 9

Other activities -0.31 below expected n = 16

Pace -0.37 too slow n = 30

Difficulty -0.13 about right n = 30

Novelty +1.03 mostly new n = 30

Read what they wrote40

Worked well9

The online presentation we got was quite good. It put a lot of the material into perspective.
I really like having two days on the same topic. Really helped with retention.
- Having two days on it, gave time for things to settle - Grounded in actual papers (Watanabe, the LLC estimation papers)
I think the discussion at the end was interesting - a lot of people asked good questions. I also had the doubt about how eliminating data is much easier to treat in Bayesian learning than SGD learning.
Kai is a fantastic teacher, Mathew was also good but felt a slight bit unprepared at times.
Lectures were clear.
Connecting LLC drops to phase transitions in models.
The online lecture -- it finally provided the megapromise for SLT showing that SLT predicts stuff that the actual learning ends up doing. This belongs to the morning of Day 1, like this should be the starting point. The lecture content was great as well, but for the improvement suggestions see above (such as should've been twice shorter). The exercises do not disappoint, they were nice, I liked them. The structure -- I think I should stop praising lecture-exercises-lecture-exercises because I have been doing that, so I'd just say keep it up. The essential points of maintaining "no lecturing before lunch" and "lecture under an hour, better 30 mins perhaps", as it was today (30 mins was great) were observed too. And I liked that exercises were given with suggested extensions, such that you could either do just the 3 core ones, or all 7. Keep that structure up.
exercises on phase transitions were nice because they were simple.

Would improve21

I would have loved to hear more about open problems and research directions. This is something that occurred to me and also applies to the other topics and lectures.
I would add coding demos for exercises.
- Some of the notation was really unfamiliar and not introduced much, a notation sheet or sticking closer to day 1 / Watanabe's notation would help a lot - Making the LLC estimation exercise part of the main day, to see the theory in practice
Too many graphs on the slide, better to have less and explore in detail. Write a bit bigger on whiteboard.
The remote contribution to the lecture did not work as well as the in person part. The material was more advanced than last time, which was very much appreciated.
I think there is probably too much focus on the mathematics in the lectures, which has generally been quite easy and posed no issue in exercises. I've found it significantly more difficult to get enough conceptual understanding, and I think the lectures should prioritise this much more. I think this debt made following the last lecture - which seemed very interesting! - quite punishing. I think many of the exercises were not particularly insightful as they felt like simple algebraic manipulations or computations in multivariable calculus. Exceptions included the exercises on why d/2 and not d and Ex. 4.2.
More context about how this relates to AI safety.
The pacing was a bit slow. Too much time was spent on basic mathematics and a few important results were just briefly sketched.
In fact, the second day of SLT was much better and more engaging in comparison to the first one. Potentially, some additional exercises on computations of degeneracies -- at least in common statistical models and explicitly seeing their implications could be really fun.
An almost 2h remote presentation, followed by discussion, with no scheduled breaks, at the end of the day was quite heavy. Great content otherwise!
It was good to get the applications of yesterday's basics. I believe some of this could be joined into the previous day to strectch the buildup of theory shorter.
I think the material should be given in a different order so that the main theory is presented (rather than elucidated in the exercised and focusing on basic definitions in the lecture). At points certain knowledge could have safely been assumed (e.g. basics of Bayesian statistics)
More practical code examples for estimating lambda.
feels like not enough new material for an entire day and the lecture times today were a bit too long to be effective
Regarding yesterday I said: "- I felt like the talks often lacked structure, which contributed for the unnecessarily slow pace; also, more content would be nice - some (not all) of the exercises were too much undergraduate Calculus-like and the "insight density that they offered" was small - also, I appreciate the presentation + exercise structure of the notes but feel like given the small extent of the presentation part, it would help to clearly separate which results I should retain from something I learn from an exercise which has a particular context; (also maybe some plots would be a nice addition)" These things still apply to today. I'll just add that the talk by Miguel was too long and the slides had too much information which made it difficult to follow. (I would have liked to hear about the papers he mentioned in a more comprehensive manner). Also, it would be nice if some of the things he presented were the integrated as a continuation of the main talk/notes material.
Adding a group activity would be cool I think
Exercises were useful, but the ratio of symbol manipulation to conceptual insight wasn't great. I think it might be possible to construct more examples that make things more visual/build more intuition. The phase transition problem was pretty good, for example. I also spent some time playing around with posteriors and V(epsilon) functions in Desmos and found it quite useful.
I don't feel that I obtained much insight from solving the exercises. I found the talk at the end a bit too dense. I found it very convenient to have written detailed notes. I think one of yesterday's exercises 2.16 was a bit confusing. It is given an example of a reparametrisation that introduces degeneracy on loss landscape. If I understand correctly, this can be made more precise in a general setting. Namely, at a critical point, reparametrization by a diffeomorphism must preserve the Hessian as a bilinear form. In the example the parametrization is not a diffeo at the origin, so it is expected that the signature of the Hessian is not preserved.
More preparation before the lecture suggested for lecturers: say running it between themselves before running it in front of the participants. This suggestion comes from the fact that sometimes Matthew was lost in what he was saying, but I can't recall his specific phrase. Smth along the lines of "Okay this actually does not belong here." I think if this is done he'll be great! For the online lecture, the soundcheck and other stuff better be run before to avoid fixing problems during the allocated timeslot. Also the lecture itself was quite long (2 hours) which inevitably leads to losing focus without a break. If we don't wanna be doing a break, perhaps a Veritasium approach (two plot lines, the movie aka lecture switching between the two to keep engagement, the trick Derek borrowed from cinematography school) would help. In fact, lots of cool stuff was in the end, where the concentration was already lost, so perhaps the talk could've been twice as short without the need to go into details early on. I'd say Julian was quite a master in this: he had like 160 slides in his MechInterp day, but it actually felt extremely engaging and not tiring at all, as each paper had one single figure from it on the slide and was covered quite briefly, with a single idea behind it, illustrated with that figure. Quite an exemplary approach.
The lectures need a bit more structure (dual support using slides is very pedagogical). I feel like these two days could have been condensed in about 1.5 days, giving time to read and discuss papers in the last hours of each day. This way, Miguel's lecture would have hit the target better and conveyed more meaningful questions and discussions.
There should be no remote presentations during the intensive. Pace can be a bit faster.

Changed my mind7

SLT has quite a few compelling ideas and it seems like it could use a lot more active researchers! But I'm still not sure how clear the link to AI safety is. Not just in terms of differential advancement of safety - I'm not sure what the theory of impact is for positive differential impact at all.
Not particularly
n/a
SLT provides a rigorous foundation for developmental stages.
became a bit less sure about if im convinced by SLT, but will spend some time doing more reading to make sure!
I realized that maybe chasing the strict minimum of the loss function landscape is not really needed. Maybe the landscapes is the wrong thing to study. The secret to generalization might be hiding in a different thing altogether (the early stopping generalization had a point, just like other methods like random deactivations). Btw the demo that, unlike GD/SGD, Adam and Muon first learn smaller components, which might be "noise," was quite surprising and exciting.
SLT during training could be interesting but needs improvement in LLC estimators.

Other thoughts3

Today and yesterday was very tough because my calculus is rusty. As a computer scientist I’m much more familiar and comfortable with discrete math
n/a
Outstanding two-day deep dive.

Abstractions and Latents Mon 21 Sep, Satya Benson 37 responses · energy 3.5/5

Overall -0.94 below expected n = 31

Content -0.81 below expected n = 31

Clarity -0.58 below expected n = 31

Motivation -1.03 below expected n = 31

Exercises -0.30 below expected n = 30

Readings -0.67 below expected n = 21

Other activities -0.54 below expected n = 13

Pace -0.29 too slow n = 31

Difficulty -0.16 about right n = 31

Novelty +1.13 mostly new n = 31

Read what they wrote47

Worked well9

Reading through the notes by myself.
having the time to read and do the exercises at my own pace
Lecturer communicated clearly.
- Reading a section and then doing the exercises straight after - The small group discussion
Speaker was clear.
Aron (?) seemed far better at explaining and communicating.
I think the concepts are very interesting and probably something I'd like to think about more in a couple of days when proposing a project!
Formal distinction between mediation and redundancy.
The structure: lecture->exercise->lunch->lecture->exercise->discussion

Would improve25

Improve the teaching and maybe the material selection
Fewer definitions and theorem proofs, more application of methods and experiment results.
To me, the connection to Alignment was a bit haphazard, contained in only a few bulletpoints at the end of the notes. I would prefer it to be more center stage, perhaps a presentation + discussion, could be very short. As it was, it felt very vague and swept aside. I found it difficult to focus during the lectures, partially because they were slow, but didn't go much beyond the formulas. I think they would be better if more intuition for what things mean (including questions like "what does this entropy tell us?") was explained.
There was no motivation at all on why this matters. I did not like the teaching style. Exercises in the first half just for working with notation and really easy.
The slides seemed completely AI generated and the concepts were not very well motivated and I think there was a shared feeling among participants that we didn't learn anything new.
Since some point of the intensive, I feel like we are manipulating the same concepts, but using different notations. I feel that there is a big overlap among courses but these frictions make it confusing and the short time that we have to familiarize with each topic everyday make it so that we cannot really delve deeper in the different theories.
There was very little content today. Most of the content that was present seemed to be information theory and very few new ideas and connections to AI Safety. It would be better if more new content was included. If there isn't much to be discussed as this is a very new field, it should be scrapped/merged with another similar session.
1. H needs to be defined with a name on the slide for entropy 2. questions were helpful in the lecture to understand the setup better, 3. in the notes label the conditional mutual information 4. add more motivation and ai safety context, how can these be tested/computed
I think this literature (which is very niche) could have been positioned better against the wider literature on latent variable models and latent identifiability in ML.
So many things. This day makes no sense. The research direction is very flimsy, the delivery was not very good and it's just difficult how this fits the alignment and safety direction. Iliad needs to seriously reconsider its curriculum to cover more in-depth established research programs like SLT.
Although the lecturer presented clearly, I think the material was not well motivated. The basic problem we were trying to solve - how two agents can agree on a set of 'beliefs' - should have been more clearly communicated from the very beginning.
- A concrete example of a natural latent early on, before the mediation / redundancy conditions - Pointing to the formal versions during the lecture (e.g. the natural latents paper) for going deeper
I'd like to see a slightly deeper theory + connections to sparse graphical models & linear algebraic algorithms to compute the things the theory cares about. At least for the simple gaussian models to get better intuitions.
The speaker was clear, but overall I think the material was not well motivated today. We should have begun with a clear description of what we were trying to achieve - i.e. how we could model different agents coming to the same set of 'beliefs' - and properly defined how the associated objects (observations, latents) relate to the problem.
I found the material very poorly motivated. The lecture exposition would have benefited greatly from a roadmap. The definition theorem format gave little insight into the 'bigger picture'
This material should be removed from the curriculum. It is absolutely not clear and not motivated why a less wrong blogpost from the last year should be taught for a whole day. Also the teaching needs to be reconsidered.
A bit more of a link to empirical things would have helped cement the information-theoretic foundations into a more practical context.
I didn't quite understand why we were learning this and felt a bit blindsided. I enjoyed the work on a technical level but still have little idea why it was useful for AI safety
I would like to hear about how alignment is done. We're halfway through the intensive and I feel like I've not heard anything about alignment
I think maybe a little more content on the algebra of information theory - the algebraic manipulations are not always very easy and even though I have some prior experience, I was defeated by the two-witness lemma.
The material felt very (nomen omen) abstract; I struggled to see how it could be applied to any more practical settings. There was also not much substance in the material, it felt like giving new names to existing concepts. Also, I think the lecturer was quite jet lagged and it negatively affected the flow of the lecture.
The talks seemed completely unprepared. There was little emphasis and clarity on why this matters. What is the big picture?? How do people use this with deep learning models? And for alignment?
Everything. This felt like an opaque subject that was not properly motivated in the beginning of the lecture. The exercises in the first part were ok to disentangle the difficult language but exercises in the second part were just "proofs left for the reader" which were not very useful to me. It would have made more sense to study real models where latents are concrete and useful. The lectures should include at least one example showing the different tree vs non-tree examples of latents like was presented in the notes. The notes themselves are better than Eisenstat's paper, but still need to be clarified with notation and language that is coherent with the rest of the Iliad curriculum. Working out an example on the white board would slow the pace down and help everyone understand. I did not follow the second part of the lecture at all. No idea what was happening.
More visual diagrams for the condensation framework.
This was too abstract, more examples needed. Not necessarily real life examples (although that is always welcome), but examples to each statement / theorem / computation would be highly beneficial during the lecture. Also, in the very beginning of the lecture we were just submerged into the subject unlubed. Having slides allows one to show the applicability or the empowering example or an impressive result or some gradual real-world examples that help us pick up the thinking so that we can translate it into formulae later. The intro does not need to be long (5-10 mins is completely enough), and I am aware an intro runs a risk of questions arising that drag the intro out into the philosophy (as the Day 1 of SLT has shown). But it has to be there to make the immersion gradual -- some people might have well woken up 15 mins before the lecture, while others, despite traveling there for 1 hour and being for 2 more hours on their feet, do need a gradual context switch. Also, the number of links that was mentioned for exercises was way too big. Like another Iliad website.

Changed my mind7

n/a
n/a
I thought the concept was very interesting and I can almost see how it ties into other topics. I would have appreciated a more holistic view on things though.
I feel a bit frustrated and want to know more about the purpose and direction of research, and would rather engage with that kind of material. These questions are fun but it feels like a poor use of time. It's hard to choose a research direction when you don't see why you're doing it. I feel this sentiment is shared, If the situation is so urgent, we should be engaging more with actual research and what is the state of the field.
Understanding condensation does not seem like something useful because we have no proof that LLMs understand concepts that same was humans do. Isn't that the whole point that superintelligent AI could be an existential risk? An example is neuralese!
Natural abstractions can be mathematically formalized.
That I probably have no wish to do dry abstract math

Other thoughts6

Have chats in person with some of us for ideas to improve the intensive. I genuinely think that its a great initiative but there has to be some structural changes to the program.
n/a
n/a
I was sick so I only attended the first half of the day
Solid foundational theory.
I will leave after lunch tomorrow because I have an onsite interview. That's why I can't make it to the London tour, sorry -- thank you for organizing btw! Must be very nice for non-locals.

AI Alignment Introduction Tue 22 Sep, Aden Power 36 responses · energy 3.8/5

Overall +0.77 better than expected n = 35

Content +0.89 better than expected n = 35

Clarity +0.91 better than expected n = 35

Motivation +1.14 better than expected n = 35

Exercises +0.30 better than expected n = 10

Readings +0.29 better than expected n = 34

Other activities +0.34 better than expected n = 32

Pace +0.11 about right n = 35

Difficulty -0.60 too easy n = 35

Novelty +0.17 an even mix n = 35

Read what they wrote58

Worked well20

I thought Aden's lecturing was great. I thought his explanations were very clear.
The suggested readings were engaging, but I would have preferred more time (eg by providing references in advance).
I enjoyed thinking about broad alignment and safety questions - it was a welcome change bringing alignment and safety front and center of the materials again
The sense of humor of the lecturer!
This talked felt very relevant for todays issues in AI safety. It would be a very different talk 6 months ago which is a good thing.
- The readings mixed classic papers (CEV, corrigibility) with current material (Claude's constitution) - The city walk
the pace of change between lectures, reading and discussion was quite fast, but I think that ultimately it's a good thing
It was a good overview of the field of alignment and introduced a lot of new ideas.
The presenter is good. The technicalities are weak.
Speaker was very good
the overall content feels very relavant to the topic of AI alignment and covers argument from both sides and rather than presenting one point of view as superior than others the points of view were all given in a fairly objective manner.
I enjoyed that the lectures strived to give the big picture instead of focussing on introductory technical details. The overview is instructive
The level and the discussions
discussions were informative
The topic itself was very interested and so was the discussion.
Best day so far. Was great to finally hear about alignment and to have discussions about perspectives on alignment. Aden was a very engaging speaker.
Aden is fantastic and his energy is very infectious. Today really helped give a bit more context too the technical stuff we did before and was very refreshing
Clear taxonomy of alignment agendas and London tour.
The lecturer is very energetic and did a very good job of motivating the problems/views, which is the main objective of this lecture.
I think the presentation of the material was great, with a good mixture of seriousness and levity. It was very evident that Aden had thoughth a lot about these things and could give general answers and a variety of perspectives on all of the (sometimes controversial) topics.

Would improve25

There was not enough time to read the suggested readings.
I think the timings? Somehow it seemed like we didn't have very much time for readings and discussion - for a few of them, I only managed to read a few pages before it was already time for lectures.
More time for reading, either by giving less reading or providing it in advance.
1. Have this day earlier in the program, 2.have more time for reading
This was quite good.
- Felt compressed as a half day, and could come earlier in the course since it frames everything else - More time on critiques of the outer / inner alignment split
On a philosophically oriented day, I think this was as good as it could be.
I think it was a great day overall. I loved that the day was discussion focused- having well-formed opinions on things like risks and solutions can only come by discussing and stress-testing these opinions with other people. There should be more of this!
everything good, that should be moved to an earlier point in the intensive
Nothing. The day was excellent.
I thought today was very good and presented material in a motivated fashion.
Was good enough today
Most of the staff was really good. Sometimes it was difficult to keep up with each reading session, but I suppose that's related to the short day. We still didn't fully connect it to any of the previous days. Some connections are easy to make, but the most interesting input from the lecturers could cover some non-trivial connections, that aren't immediately obvious.
Minor quibbles on the slides: I don't think the definition of Inside vs Outside View was the same as the standard usage, the hand-wavy equation used to explain Inductive Bias was perhaps fuzzy enough to be pedagogically anti-helpful. On the big-group discussions: I liked that there was lots of back-and-forth during lecture. However, a good chunk of it seemed to quickly lose sight of the topic at hand. I felt like some fraction of the things people said were either (a) intended part of a different, non-alignment-related conversation or (b) so vaguely phrased that I couldn't understand what was meant or what role the statement was meant to serve in the discussion (Observation? Rebuttal? Thesis?). Perhaps it's possible to find a way to guide, shape, and clarify discussion without stifling it? I don't know of any magic bullet for making this happen though. On small-group discussions: Seemed like people mostly didn't finish the readings with enough spare time for a good discussion.
I thought this day was run particularly well.
I think today was nice in general. Maybe a bit more of time for reading on our own.
Clickable links in the notes to the different articles that are mentioned? Save lecture notes in a different format than the beamer so it's more pleasant to take notes and follow along.
Nothing. I really liked it today
Best day so far. Was great to finally hear about alignment and to have discussions about perspectives on alignment. Would've been nice if this day came in week 1
Could've been a bit more time for the reading
More time to discuss threat models.
Having this as the first lesson in the curriculum!
Maybe a little more time for the reading could be good. I barely managed to get into the texts by the time we were asked to discuss, especially when I asked Aden something after the lecture (and before starting the readings).
the slides could have been a bit better, but the teaching was great
move this to week 1 day 1

Changed my mind10

I do not think an AI agent cannot be goal directed.
It is surprisingly hard to define alignment.
n/a
I need to read up more on how people think about alignment, not just about the concrete technical things being done to understand the capacity of the current models but about the what the future directions would look like, as an extrapolation from observations we've already made about the current models. because the current models will be the weakest the models will ever be..
Reasons for pessimism at least feel more salient today than they did yesterday, although this may be more a change of mood than of beliefs.
I gained some vocabularly for some of the more philosophical aspects of alignment.
Learned a lot about current state of the art and community beliefs about AI alignment.
I am still unsure if there is anything we can do at the moment, but the point that people in frontier labs are (perhaps)reading alignment research gave a bit of hope for the future
Sharper distinction between capability vs alignment targets.
Recently I have been more worried about the misuse aspects of AI, and today reinforced my concern about alignment, mostly because of the discussion around instrumental convergence.

Other thoughts3

n/a
I left after lunch because I wasn’t feeling well
Tour was a great break before proposals.

Project proposal day 1 Wed 23 Sep, Leonard Bereska 29 responses · energy 3.7/5

Overall +0.19 as expected n = 27

Content +0.04 as expected n = 27

Clarity +0.04 as expected n = 27

Motivation +0.15 as expected n = 27

Exercises +0.80 better than expected n = 5

Readings +0.67 better than expected n = 6

Other activities +0.52 better than expected n = 21

Pace +0.41 too fast n = 27

Difficulty +0.37 too hard n = 27

Novelty +0.07 an even mix n = 27

Read what they wrote41

Worked well13

I liked discussing with others. I appreciated the feedback from the fellows.
I really enjoyed talking to everyone and hearing their ideas, and trying to review the literature with the help of Claude helped me learn a lot about what is the state of the art in the field im interested in.
I think it was an interesting and valuable experience to figure out how to write a project proposal!
I thought today was a great chance to think about research directions and ideas.
I think this was very useful practice. I think the next proposal I write will be much better.
The feedback and discussion with a fellow
I think the general idea of the day is very good. The feedback loops are also useful, but I didn't get much feedback as only one person signed up to give me feedback.
Working in groups.
Vibecoding to test toy models for our proposals
Sitting in groups and discussion research ideas was quite helpful.
- Comments from mentors and fellows on the doc during the day, really useful for tightening the question
The schedule given in advance, which really helped the time management. Also the document structure in general -- was nice to be able to see what peers are doing after I finished with mine. Meaning, it is better than random allocation between reviewers and proposals (I would've died reviewing Abstractions and Latents ones).
Getting feedback from other participants was useful. It was good to get feedback from people with a similar knowledge base as myself.

Would improve20

I didn't really like the structure of "the team has to produce as many proposals as many members it has". I had expected the lecturers from previous days would be there to give us feedback - this would have made the day much better.
I found the whole thing a bit difficult, mostly due to my own lack of familiarity with the topic and desire to do something interesting and impactful. Due to the different knowledge of each person, it often made it hard to synthesise a good proposal where each person could contribute something meaningful. This is not so much a criticism of a structure, but more a point about the difficulty of running days like this. I think splitting it into two days that are closer together, would allow for more 'settling' topic wise, as in the end I felt in a rush. Leonard was a bit unclear on if we all propose the same project but in our own works, or if we all make individual proposals under the same umbrella group. As for learning papers in my area that I would build from, I found it frustrating to just have to trust claude summaries, I need about half a day to really read a paper, and I believe that at least taking that very small amount of time to digest something gives far more insight about what they have done and what is left to be done. I feel like although I understand the maths of everything with relative ease, understanding it on a deeper level takes slightly more time. But again, this is also a criticism of my lack of preparedness.
I found it a little difficult to focus with the density of conversations in the room - maybe it would have been good to reserve a place or time for quiet work. I enjoyed discussing ideas with other people during the day, but I worked on the proposal by myself as I simply thought it would be much more difficult to create >=N proposals in a group of N for N >= 2! Lastly, while I was pleased with my proposal, I think it's difficult to come up with a good original idea in one day - it feels like most have already been researched by the professionals.
I thought the day was great, however I never got the chance to speak with a mentor. If mentors could make sure to spend just 5 minutes talking to each group/person doing a project that would have made it even better.
I think the directions of what we're supposed to do could have been more clear. Especially it was hard to understand what is expected of us in terms of working in groups or individually.
More fellows to help throughout the day
Longer word count for some parts on form
A bit chaotic. It's very difficult to write a proper proposal in one day, specially, considering that this topics are not within our expertise. I think some initial suggestions/questions already given would have been nice input, even if in the final proposal we diverged. Otherwise, its difficult to make a sensible question that can be expanded in a day.
Add more structure, that was a bit too chaotic.
I think I would have benefitted from some additional structure and/or time. I found it very stressful, the way it was. This could include starting to think about this on an earlier day, for example choosing a topic. Some more concrete information on what to expect on the day would also be good :)
Maybe a little bit of more structure around how to organise groups.
I thought today was enjoyable. Perhaps make it more structured in the future.
It would have been better to have some more time to discuss with people about the feedback
This exercise needs more guidance and structure. A list of open problems per research field should be supplied. TAs should circulate and engage in conversation with participants throughout the day (not wait for participants to go to them). Saying stuff like "some research proposals are too good to be true" is counterproductive without constructive feedback on what is a realistic project proposal. One way of preparing this exercise in advance would be to present open areas of research at the end of each new subject.
- Share the submission form's sections and word limits at the start, the doc template was different so I had to restructure at the end - An example of a strong proposal would help calibrate
More time for feedback, and more guidelines for reviewers (what to look for, how specific a comment should be). I did my best in providing actionable comments (that is, leading to edits in the documents which I believe would improve it): however, I feel like I lost the reviewing lottery since reviewer 1 left a generic approval paragraph, and reviewer 2 communicated nothing. I'd question the default to group work for project proposal drafting. I concede that one reason to prefer solo work might be over confidence, but the opposite might be true: personally, I had to carefully check papers before coming up with something, and did not want to waste anyone's time dragging them in a possible aimless chase.
I think specifying that people should use the structure provided in the Google Doc for the pre-feedback iteration, and the structure given in the submission form only later, would help. Or was it not the intention? Then would be nice if that was specified explicitly. Also perhaps a good idea to give a heads-up that the proposal would have word limits in different sections, so that people do not formulate a lot from the get-go. Also, the document logistics should be more clearly specified (I did not notice the "Feedback" sheet where I had to put my paper). Also, being able to book your name there before going and providing the feeback would be nice-to-know-beforehand. Also, what might've been lacking was perhaps a suggestion that a proposal should not be too general or ambitious etc.
The grouping and feedback structure were pretty confusing. Also, I think presenting our idea actually could've been useful; maybe pair groups up and have them present to each other?
Would've been nice to have more senior people to brainstorm with. A structured morning where we split into groups with a fellow/lecturer in each group would've been nice
I found today to be particularly difficult/stressful. Having one day to create a proposal in an area I am not familiar with is asking quite a lot. The limited time and background knowledge strongly pushes one in the direction of being overly-reliant on Claude/Chat-GPT to conduct literature searches and identify open problems. I was not comfortable with putting my name on something I did not have enough time to develop properly. The format of the proposal was also prescribed, and appears quite unnatural for me. Having a suggested format would be fine, but it would be good if the proposal format was more open. A summary of my motivation, interests, and thought processes is perhaps more appropriate than a specific proposal given my background at this point in time.

Changed my mind6

I do firmly believe that doing this research is important and necessary, and will use my next few months to try and understand the current state of the art better. I would be very glad to sit with a project which may in some way help our understanding of AI and learning. Obviously I have my own ideas about what would be effective or not, but regardless, I feel sometimes the culture of this place is a bit odd to me, the scale of the funding available and how insular it can appear is something new, especially coming from academia. I had some conversations with other people here and I asked them if people (and research groups) are just using the funding to extract what they can from the field while its hot, well aware that other industries may collapse in the future. Almost everyone said yes and said that most people here would probably take a job at a frontier lab if It was on the table. I'm not necessarily against this for others but it was an odd thought to grapple with. It made me question if it was in some odd way selfish being here. I guess it made me question why im here, why I feel unwell seeing some aspects of the culture, if I want to fit in culture so that I could contribute to the research, or if I want to play some game to ride a wave of funding as far as possible. I think almost everyone here is ready to work towards the goal of AI safety, but there is also the normal urge to find a pet project and just do something cute and find a hole in the research so we can just pump out some papers and be in some glass tower. Maybe thats a bit harsh of me im just venting. Regardless, I think the work is super important, I've already registered for an SLT conference next month and will attend, and will start looking for projects to do in the coming months.
Could imagine doing a project or fellowship in a better way now
n/a
I realized I am able to do some experiments that might be relevant for some areas of AI safety.
Coming up with smart research is hard.
That writing a good proposal is challenging, and is a fine balance between smth specific and impactful.

Other thoughts2

n/a
It was a good activity.

Agent Foundations Thu 24 Sep, Aden Power 28 responses · energy 3.8/5

Overall +0.50 better than expected n = 26

Content +0.35 better than expected n = 26

Clarity +0.27 better than expected n = 26

Motivation +0.62 better than expected n = 26

Exercises -0.42 below expected n = 26

Readings +0.08 as expected n = 26

Other activities +0.24 as expected n = 21

Pace -0.12 about right n = 26

Difficulty -0.31 too easy n = 26

Novelty +0.96 mostly new n = 26

Read what they wrote37

Worked well12

I bet the game would've been cool :(
Emphasis on group discussion and activities.
Activities and general pace were great! Loved them.
Aden is a charismatic talker and follows a clear line of reasoning/motivation in his teaching. Very good.
I think Aden's lecturing was again great.
- Seeing several of these core ideas/ agendas side by side made it clearer what each one is trying to formalise and how they relate
The game at the end was fun- something different!
Aden's explanations of Löb's theorem and embedded agency were fantastic.
Lecturer was very good.
Enthusiasm, explaining the different research agendas
The presentations were very clear and well structured.
I did like Aden's enthusiasm! Also the freedom to choose what paper to read was very good, as well as exercises for each of the 5 research directions. The game was a fun activity as well.

Would improve15

Today's exercises used a lot of notation to describe relatively straightforward concepts. The problems themselves also weren't as focused on teasing out the core ideas as I might've hoped. For example, proving that a binary search converges didn't add much to my understanding of why any agent that can respond appropriately to certain changes in the data distribution must've learned an implicit causal model.
Maybe some preliminary reading would have helped.
The reading for garrabrant induction was great but the exercise was bad and you should have put in the effort to have one awake and focussed human read it
Overall, everything was great. It's definitely a bit dense in material, and I'm not sure I got every detail, but as a high level overview of the very broad Agent Foundations agenda, the day appeared to be great. A bit meta, but over the course, I get progressively convinced that people need some external organization and splitting people into separate rooms with whiteboards and a list of topics that are connected to read material would be better than allowing them to self-organize or work alone. At least finding such rooms and encouraging everyone to use them (groups of 5-10(max.) people would be great).
I think more time spent on the mathematics of agent foundations, especially in the lectures, would have been good.
I think the exercises were interesting. I would have liked to have time to work on other exercises from the other proposed topics. Today's lectures were refreshing with respect to other days, I think I enjoyed it more than most of other sessions.
- A lot of ideas in one day! (coherence theorems, Löb's theorem and tiling agents, logical induction, decision theory). Going deeper on two or three would help - A short refresher on the formal logic before Löb's theorem, for people without a logic background
A bit of a muddled up day. Some parts were good but it was a bit all over the place, it seemed like three lectures in one. Everything was claude generated as well- this is a pity and I actually quite dislike it.
The exercises were a littel hard. More concrete feedback on the specific feedback form
Nothing, I enjoy the day throughout
TV screen for slides is too small to see (23 curtain screen was larger); hot water machine broken (no hot water).
It was nice in general!
I thought today was very good.
I would add a coding demo for one of the topics.
I don't know if it was just me, but Aden might be too loud at times. But it might've also meant that my sensitivity spiked. When it was the last lecture on cooperation, I think clarifying that we do not consider empirical results or actually mentioning them for completeness (such that in a Prisoner Dilemma tournament tit-for-tat strategy, which is cooperative by default, won) would help so that no one else raises it or points it out. The day felt very abstract, which might be the field's feature rather than the lecturer's. The game was lacking in interpretability a bit. Discussing it in more depth would've been cool. The first lecture started very strong and I was so engaged, but once we started proving a theorem rather than just going through the main statements (proof sketch), I was lost -- for me it was engagement breaker. THe first lecture was also way too long. If I remember correctly, it lasted for 1.5 hours non-stop.

Changed my mind8

I have less confidence that the entire field is on solid ground.
In fact, I was more or less interested in game theory for quite a long time, but struggled to find time to dig deeper. Although I think that theories that treat people as perfectly rational agents are confusing, they must work much better for future LLM agents, even not superintelligent once, and having a good theory of safe improvements could allow to handle more problems and negotiate on a wider range of topic successfully. Besides, advancements in game theory have presumably saved lots of lives before, and allowed politicians and stakeholders to build better models for negotiations. I tend to believe that it might be one of the most promising research agendas in the nearby future.
Ironically, I am more and more skeptical of theoretical frameworks as the intensive progresses. I came in with a lot of enthusiasm for theories (I come from SLT and was expected similar things) but now I am more and more convinced that the way forward in patchy work like mech interp.
Much clearer perspective on safe self-modification and Vingean reflection.
Discovered applications of Game theory for agents foundations, so I could use my GT background in this direction
n/a
Decision theory is quite nuanced
That mathematicians and whoever works on all of this are "the drunkards searching for keys under the lightpost cause that's where the light is." It seems like once upon a time, a mathematician relaxed the infinite compute assumption and AIXI immediately manifested and killed them. Why is there so little work on inconsistent preferences and arguments beyond Dutch Books and Money Pumps? It honestly makes me furious -- no one money pumps anyone, and the theoretical possibility does not mean that someone will appear and do that, as that requires trying to find the inconsistency in another agent's preferences and then make sure that the agent would not notice exploitation (and he almost surely will, as he is not stupid despite inconsistent preferences). Logical Induction was the only promising thing out of everything taught today. No one maximizes any utility function, that's a myth, and for that to be happening agents have to have infinite compute / time and that is never the case. No human does that, even if they claim they do, because there is no way they can execute those operations on made-up probabilities and payoffs that do not take a lot of things into account. Aurelio raised nice criticism which was similar in spirit, and it was nice of Aden to respond by two arguments: 1) We study this so we do not reinvent the wheel, as oftentimes people wonder "what if the agents works like this?" and he goes "well, we already thought about this" and 2) So that we can understand the future machine that we will build. But argument 1 is weak because you can just tell people that their assumption is unrealistic instead of launching an entire research program preemptively, while argument 2 precludes us being killed by an agent with inconsistent preferences before we get to build a consistent one. I believe the field should abandon all the convenient math and introduce realistic assumptions (bounded compute, polynomial time etc) and work with them. Otherwise it's waste of math.

Other thoughts2

Overall great day.
n/a

Alignment in Practice Fri 25 Sep, Aden Power 35 responses · energy 3.5/5

Overall +1.06 better than expected n = 32

Content +1.25 better than expected n = 32

Clarity +0.84 better than expected n = 32

Motivation +1.44 better than expected n = 32

Exercises +0.63 better than expected n = 8

Readings +0.14 as expected n = 22

Other activities +0.11 as expected n = 27

Pace 0.00 about right n = 32

Difficulty -0.13 about right n = 32

Novelty +0.50 mostly new n = 32

Read what they wrote55

Worked well15

Great teaching!
I think breaking up the lectures with the game/reading/etc was great, overall just a great day. Aden is an incredibly entertaining and competent lecturer :)
- Walking through the pipeline phase by phase (pretraining, post-training, deployment) gave a clear structure for where each alignment method fits
Practical overview of post-training affordances and empirical failure modes.
Everything landed; best session of the programme
The lecturer's enthusiasm and a lot of real-life examples + rumours and reports about real companies and state of the world. Aden added a lot to the lecture in this way, by telling about firms etc. Very grounded in reality lecture, I loved it. Resembled Julian days 1-2 in many ways. I also loved the Forbidden Techniques discussions, those were extremely informative. As well as train / val /test sets.
Great energy from the speaker!
it was great
The small game we played was surprisingly instructive. The discussion about the progress of the "practical" alignement field was very important for me to understand the concrete steps being taken at the moment, definitely a core lecture module for the intensive. Kudos to the teacher, who managed to remain engaging and clear through a very long lecture day!
Aden's lecturing was excellent once again. The content was chosen very well too. Very grateful for the effort Aden must have put into the last few days - thanks so much.
The content was amazing. Felt super relevant, it was clear why its important and how its useful. It was also amazing context. Wish more days had similar content.
I think the presentation struck very good balance between technical details and general ideas.
Motivation with examples.
Very relevant content and I can tell the teacher is very passionate + knowledgeable about the literature and developments of the field
The speaker.

Would improve23

I would need more breaks to follow the entire day
Maybe some discussions ran a bit off-topic, but it wasn't too bad at all.
- More time on post-training (RLHF and the methods after it) as a proportion of the day, since that is where most current alignment work happens - Links to the key empirical papers mentioned during the day, to read afterwards
Readings didn't add a ton beyond what was already explained in lecture, or perhaps it wasn't clear what we were supposed to get out of them/which parts to focus on.
Nothing.
More time analyzing concrete lab safety cases.
Nothing comes to mind
It was cool, but it was a bit too much talking in one day. It can be split in two
Add breaks in between 1.5-2h lectures. Also, I'd perhaps decrease the information density delivered vocally. During the lecture today, I was recalling Julian's lecture on Day 1 in MechInterp, which was exemplary in many ways. The slides were very informative and with optimal amount of info -- one figure per paper, few words per slides but many formulae because that's what belongs there. Here, on Day 15, the slides are as many (100+), and they are informative, but we do spend quite a lot of time on each I believe, which destroys pacing. Slides should be changing more often, I think.
Awesome day.
The lectures were really great, but I think a couple of more breaks would have been appreciated.
maybe one more short break after lunch would have been nicer
The lecture was very good and very well motivated. The framing of part of the content was also somewhat "historical" (i.e. following the evolution of the field). I have found that to be very helpful, and this perspective may be worth to stress further through the material. The lecture also has many short examples of techniques being used and the ideas behind them, this can be a but overwhelming at times, but also very interesting. Maybe cutting 1/4 of the examples and expanding the others would be good (the rest can be given as reading material) Overall the lecture was very successful and I would not change much!
I think, just in terms of pacing, it was difficult to maintain focus for multiple lectures of this length, despite the fantastic lecturing and content. It being a Friday may have contributed! - maybe this schedule is better suited to earlier in the week? The game today was maybe a bit too simple to feel meaningful. I struggled to follow the section on GRPO a bit.
Today was the best day so far!! Loved the content. Would've loved for it to be split over two days to maximise retention since the content was so so important.
This day was awesome but could've easily been to days. We had days with a third of the content here.
Do breaks after one hour, no 2 hour or 1.45 hour lectures
I really enjoyed today bcs all topics are super relevant. however for my taste there was too much lecture/ passive engagement and not enough discussion/ active engagement. it might have been nice to distribute this day over 2 or 1.5 days and add more discussions in between.
It was a very interesting day but way too long. We sat for like 6 hours of lectures and however interesting Aden is that is simply too much. The content is very useful for people (although broadly irrelevant if you've done any applied alignment stuff) so maybe it's worth splitting it.
more discussions on whether other types of persona could be useful for us + how we expect continual learning to look like / rough predicted timeline on when it would emerge
I thought today was the best day. Aden was a very good speaker!
Nothing. The day was fascinating.
Lectures were a bit long and I found myself not following at different times. Maybe switch with readings a few times during the day?

Changed my mind13

Learned about some ai safety hackatons at ARC
More differentiated understanding of risks
Slightly increased optimism that, given cleverness and perfect implementation, we can find adequate ad-hoc alignment and control strategies for another year or two of scaling.
Clearer view on how RSP commitments translate to technical evaluations.
Safety cases feel more tractable than I assumed
I think it stayed somewhat similar to BlueDot Technical AI Safety, just way more informed. It does tie well with some of the reasoning I was doing regards train / val / test sets as well, was very useful to hear about it again.
i got convinced that doing safety finetuning directly on internal activations is not a good idea
I think today's lectures made me more pessimistic about the current state of affairs than I was previously - but it is probably net positive that I'm more accurately calibrated now.
Frontier models can get out of control very soon
still not sure how to think about evals
n/a
Control techniques are more diversified than I previously thought
I will not use the forbidden technique

Other thoughts4

Very engaging and practically relevant.
I can't find the MechInterp slides, could you please point me to them? Neither on Discord nor on Github nor on the websites
How are you interpreting these answers? Like, the pace was quite fast for someone who's never seen the material but it's tautologically slow for someone who's already seen everything. How people interpret the questions will vary, so answers should be interpreted conditional on how you think most people will interpret the questions. I think you might get more information overall by having 10mins chat with each of us individually to ask questions. I have a lot of ideas (and concerns!!), and I know the same is true for many other people.
n/a

Debate Mon 28 Sep, Stephan Wäldchen 26 responses · energy 3.3/5

Overall -0.04 as expected n = 24

Content 0.00 as expected n = 24

Clarity -0.21 as expected n = 24

Motivation +0.08 as expected n = 24

Exercises -0.43 below expected n = 21

Readings 0.00 as expected n = 23

Other activities -0.11 as expected n = 19

Pace -0.04 about right n = 24

Difficulty +0.08 about right n = 24

Novelty +1.21 mostly new n = 24

Read what they wrote36

Worked well8

The game in the beginning -- incredible warm-up, where we were looking for mistakes / errors in simple proofs. This is how most lectures should be starting honestly. Nice balance between exercises, lectures, and papers. Going through exercises together! This was the first time it was done. Delegating it to going through the solutions perhaps also works, but definitely not as good. One of the best days I had indeed -- very applicable math, which finally goes into computational and memory complexity from applicable perspective -- our agents no longer possess infinite compute and memory. This is a great start. As if all my grudges from previous days (where fields had unrealistic assumptions) were finally addressed. And overall the day was very practical, grounded, varied, and at the same time fundamental. Loved it from beginning to end. Lecture on modeling beliefs via graphs -- good, very good, very empirical and theoretical at the same time, Econophysics vibes. Also, Stephan is great lecturer, excellent in fact. Or maybe I am biased to the topic, but really, I was engaged more than most of days.
Speaker was good.
The reading and discussing was separate but also intertwined well.
Reading by our own
Nice choice of reading!
I think this is a really interesting field and one I hope to revisit outside of the Intensive hours - I can definitely see myself working on this further, and I might try to write about it for my project proposal.
Complexity-theoretic framing of debate and interactive verification.
Reading papers and discussing results as a group was quite helpful.kept things very concrete in the second part of the day . Talking about what the curent state of research is.

Would improve18

I would perhaps explain first the idea of applying debate to AI oversight (namely the last presentation). I also found the latter parts of the lecture (related to statistical applications of debate) more interesting than the deterministic debates.
We really need a paper-reading workshop. We had like 30-40 mins per papers, and that time flies. Otherwise can't point out much, I liked the day.
Give more context for the problem at hand.
Some of the reading was not very high quality.
I think it would've been helpful to have more directed readings. Given the limited time available, I'd prefer you offer some hints as to what parts of each paper we should focus on, what information we should try to get out of the paper, or why the paper is important or surprising. The exercises were fine but somewhat redundant with what had been explained in lecture. I'm not sure if it's worth changing them though, interesting problems are fun but not essential for learning.
The narrative is a bit unstructured and hard to follow, and it seems the lecturer wasn't aware of the material that already was taught during the course. It's not a significant problem, but as the lecture is on the last week, contextualizing it better is definitely possible. Some results from AIXI, abstractions etc. are directly applicable to debate.
I prefer slides, but this is something personal
Good lecture day, thanks. I would suggest much clearer planning: for example, trying to adhere to the schedule, stating time allocated for reading. Also, more time for reading was needed, in my opinion. It felt a bit wasteful, after a break, to recall everyone in the main lecture area, waiting for them to be seated, only to announce we had to go back to our seats and read.
I think the content was generally good. The exercises at the top of the day didn't seem very related. I found the content a little demanding today, so in the afternoon, I preferred to read through the notes/papers myself rather than listening to the lectures, where I was starting to get a little lost.
I think doing exercises together on the screen is not super useful, since the amount of time people need to think through them can vary a lot, depending on background. Introducing the AI case a little bit earlier, even just a few sentences during the introduction could be nice as well.
My main criticism of today has to do with (what I felt to be) the lack of lecture time: the lecture parts could have been longer and more formal (but still comprehensive); the little formal content was in the reading/exercises, which for me is not preferable in comparison to the average modus operandi of the previous weeks. I'm not being exhaustive about the things I liked, but I definitely have to emphasise how the way papers were discussed (all together and with slides) is a huge improvement compared to previous days.
The presentation is a bit too math/cs, I think that currently applied debate is different and doesn't use all that machinery. So it may be restructured, e.g. going with practicalities first, and then give theoretical underpinnings. Also, using the latest papers might not be the best idea because a lot of them are slop and better look for some battle-tested blogposts even (so the reading is not well chosen).
I would improve the choice of papers for the first session. They were rather bad papers and were not worth my time.
More motivation at the start, some time to do the exercises.
More time on prover-estimator debate variants.
I found the notes on the complexity classes a bit strange. They try to be introductory but at the same time have to make compromises and present more advanced topics, making it overall a bit confusing. I think having slightly longer notes for the sake of complteness would be very nice.
I don't see how the experimental results connect to the theory except that there's some justification provided by the theory to do debate.
The first part of the day could benefit from having slides. I found that the discussion about Turing machines was a bit confusing. The exercises and notes would benefit from clearer language and a few reminders about complexity theory .

Changed my mind6

That Debate might be extremely effective. I recalled that it incentivizes each side to pull the most extreme arguments, which definitely helps. Also, modeling beliefs via graphs was mind-blowing -- simple yet realistic and effective. We need more theories like this.
n/a
Slightly more optimistic about usefulness of debate
learned about AISI
Debate offers a much more credible path to scalable oversight than pure distillation.
I found the contents very appealing for research.

Other thoughts4

I will have a few interviews tomorrow and will thus likely go to Honeywell to take them, but it will be 1h between 3pm and 4pm, thus shouldn't be too distracting.
n/a
I had a terrible day for personal reasons and it was very hard for me to pay attention today. The feedback reflects on my experience but it is by no means objective.
Super clear exposition of AISI safety cases.

Steganography and Backdoors Tue 29 Sep, Stephan Wäldchen 23 responses · energy 3.4/5

Overall +0.05 as expected n = 21

Content +0.29 better than expected n = 21

Clarity -0.10 as expected n = 21

Motivation +0.43 better than expected n = 21

Exercises -0.05 as expected n = 21

Readings -0.10 as expected n = 20

Other activities +0.52 better than expected n = 21

Pace +0.24 about right n = 21

Difficulty +0.19 about right n = 21

Novelty +1.10 mostly new n = 21

Read what they wrote29

Worked well10

The speaker was clear.
The game was super fun, the reading were very relevant and the exercises very illuminating (ie. the relu exercise). Excellent format and content!!
The game -- great warmup and the way to taste the subject with your own brain. This helps immensely. The exercises -- very physically coded, so I am biased.
Game was fun.
The steg game was great :)
The game
Reading then discussing
I think the structure of the day was good and I liked Stephan's lecturing.
Information-theoretic bounds on steganography H(M) <= H(K) and Merlin-Arthur verification.
The game was quite good.

Would improve12

Present more motivation for the problem at hand; e.g. begin with a cursory overview of cryptography, and how this differs.
This was an incredibly interesting day. There seems to be a lot of empirical stuff which would benefit from more theoretical contributions (ie. a la Shafi). I would really split this into two days.
(Same as yesterday:) My main criticism of today has to do with (what I felt to be) the lack of lecture time: the lecture parts could have been longer and more formal (but still comprehensive); the little formal content was in the reading/exercises, which for me is not preferable in comparison to the average modus operandi of the previous weeks. I'm not being exhaustive about the things I liked, but I definitely have to emphasise how the way papers were discussed (all together and with slides) is a huge improvement compared to previous days.
The game -- it was fun, but lacked structure and general advice. We overthought a lot as well. Also, more work on explanations would be nice.
Similar comments to yesterday on the approach to readings. I liked today's exercises more than some previous ones. I thought the steganography was relatively straightforward, whereas the backdooring was more interesting/surprising to me. So maybe I would have liked reallocating time to go deeper into the latter.
Maybe do two rounds for the steganography game and give more time, or smaller teams? We had a bit of trouble to get organised. Should there be a pedagogical lesson from playing the game? Some truth that should be revealed from playing? Maybe say how dangerous doing research in this area is and how to mitigate the risks?
Today it was a bit exhausting. Some of the slides where heavy in formulas and it was difficult to retain the meaning of the symbols in between slides. I think the content was great, i particularly liked that it was more concrete that what we have done other days of the intensive.
number and define equations with all their quantities on the exercise sheet , better to have more time for some reading and in group discussion than more and everything rushed. I would prefer more of this day and less debate.
So many different presentations and topics, so it was difficult to follow the whole day.
I think the material is good, and I enjoyed the lecturing, but maybe the structure? It sometimes felt more difficult to grasp some concepts than I think it should have, and I found it difficult to work out why, but I think it was mostly the distribution of time per idea - too much for some ideas and too little for others.
Hands-on implementation of undetectable backdoor constructions.
Everyone's a bit tired so I think this day could've been a bit lighter on lectures and more discussion, reading focused.

Changed my mind5

n/a
Yes!! I definitely want to work more on this kind of workk.
It made me more concerned about steganography.
discovered some labs that take students without a master, that work on safety
Paraphrasing defenses are far more fragile against steganography than assumed.

Other thoughts2

n/a
Fascinating module, really sharp math.

Agent Foundations Wed 30 Sep, Ashe Vazquez Nuñez 27 responses · energy 3.5/5

Overall -0.20 as expected n = 25

Content -0.12 as expected n = 25

Clarity -0.04 as expected n = 25

Motivation -0.36 below expected n = 25

Exercises -0.25 below expected n = 8

Readings +0.17 as expected n = 24

Other activities -0.05 as expected n = 20

Pace -0.28 too slow n = 25

Difficulty -0.08 about right n = 25

Novelty +0.92 mostly new n = 25

Read what they wrote35

Worked well8

Ashe is very enthusiastic and thoughtful, and engaging. That helped a lot.
I think I had high expectations which weren't fully realised, but the day was still good overall. I particularly liked the 3-person discussion.
I liked the philosophical discussion at the beginning of the day
The readings and the free flowing discussions.
Intuition for updateless decision theory and reflection.
I read about open-source game theory, which was interesting.
The speaker was clear
i liked the reading about logical induction

Would improve20

I liked this approach very much- I appreciate that it has tried to link it back to the content covered during other days.
Maybe it should be more broad and more impartial as in less lesswrong-coded, there are a few research directions that fit "agent foundations" idea well, but don't have a corresponding lesswrong post. As Iliad reaches out to people from academia and other backgrounds, it would be nice to accommodate them as well. Know thy audience.
Maybe a bit more of an overview of agendas would be nice, some broader classification perhaps, with blindspots. As the day seemed to be mostly driven by ourselves and readings. Also, the proof in the end pertaining to Intrapersonal game theory took much more time than was available to it, that needs to be fixed.
I see the value in this day, but it's hard to engage with these topics for three days (built upon earlier agent foundations that were similarly philosophical and qualitative). In particular, it is unsatisfying (as part of a 1-month intense programme) to have this many days at that level of intensity (after all, philosophical discussions are tiring) but not have some measurable progress. On the other hand, there were topics like steganography and data attribution that needed more than one or half a day to get a good handle of the mathematics upon which these fields are built. I would have liked to get a better feel for such topics, while contemplating the broader context of the work they entail in the background (as is common when carrying out any scientific research).
The come-up-with-a-project period was fun, but maybe not optimally structured for getting the most out of it. I spent a lot of time skimming various papers and aimlessly musing, didn't get very deep into anything.
Nothing, it was a great day. Perhaps, a more thorough introduction to the various research agendas would have helped.
As usual, I did not have enough time for the reading parts.
there was so much freedom. it would have been great to have a more directed teaching and program. The topics where sparse
It felt a bit hand wavey at times
I thought the day could've had more structure. The readings were very good but I didn't quite tie together for me.
Maybe would be better to bound more the agenda activity. The goal of the activity EAS not very concrete. Having discussions with other participants was great. It was less intensive than other days.
More time on logical induction trade-offs.
It felt like there was very little teaching today. I didn't find the overview at the top of the day to be very useful. The readings were fine, but reading can be done anywhere at any time.
This day could have had more time, especially given that we had three days for agent foundations, which was far more philosophical rather than quantitative.
In my opinion today needed more teaching and less reading / project development time. Most of us are seeing this stuff for the first time, so I think it's ambitious to expect us to be able to develop theories in new and interesting ways - we first need to learn and understand the material better.
I did not find the final talk was very pedagogical. It needs to be modified to make more pedagogical sense. It was too narrow. The "make your own agenda" is hard with the time we had. Agent foundations days should be back to back. It's the only way to make the "make you own agenda" activity make sense.
I found it too philosophical and not formal enough
It seems the day was little empty and lacked taught material or exercises. Given that the first day on Agent foundations was mostly theoretical, it'd be nice to derive some properties and maybe even see some models / dynamics in computational scenarios that can't be derived with pen and paper. However, maybe accounting for the fact most people are tired is also a good strategy.
actuall following in schedule more strictly and bridging the gap between very general statements and concrete theorems that we can propose and prove about agent foundations - in agent foundations day one we already went through readings with fairly high level of abstraction, so i think more concreteness could help.
I was a bit happy about "make your own agenda" as an exercise. The reason is probably that I am unhappy about the pre-paradigmatic state of AI alignment, where there's no established research directions (with established results and conjectures, senior and junior researchers, and open questions), but instead where everyone cooks his own soup and there's little feedback mechanisms and a lot of re-inventing the wheel. Maybe the exercise could have talked about this? This might also not be super specific to today, even though today's exercise triggered it.

Changed my mind5

Decision theories are much more fundamental to safety than expected.
It's interesting that MAISI is working on open-source game theory.
n/a
I updated my perspective on Open-source game theory, and I'm increasingly convinced it's one of the most important research directions (or maybe some of its extensions or modifications) to pursue. It won't solve alignment, but will definitely help in international agreements, illustrative examples and building the foundations for the agentic economy working at superhuman speed (the current one is kinda broken)
time is an important dimension to think about because not all timescales are equal for all agents

Other thoughts2

Superb continuation of agent foundations.
n/a

Data Attribution Thu 1 Oct, Louis Jaburi 15 responses · energy 3.5/5

Overall +0.14 as expected n = 14

Content +0.36 better than expected n = 14

Clarity +0.07 as expected n = 14

Motivation +0.43 better than expected n = 14

Exercises 0.00 as expected n = 12

Readings +0.38 better than expected n = 13

Other activities 0.00 as expected n = 4

Pace +0.14 about right n = 14

Difficulty -0.07 about right n = 14

Novelty +1.14 mostly new n = 14

Read what they wrote23

Worked well6

Speakers were clear.
The materials are very clean
Paper selection -- just incredible. As well as exercises and concepts. Very physics-coded
the exercises were instructive and let us go over the main concepts.
I think the lecture notes were written well and the material is compelling. I just would have wanted more time to think about these ideas!
Unifying influence functions with singular learning theory.

Would improve11

Share slides in during presentation or in advance so they can be annotated. Liked the chain diagram in presentation Feedback on Notes In section 1.4, is xi input and zi output, what is d_p, what is supposed to be p ? Missing y in “This question is typicall split into two parts:” in notes Some of the equations at the start of section 2 are not labelled.
It felt like there was actually no teaching and we had to read everything ourselves. Might be related to the fact that the speaker was online.
Interesting, but more lectures would have been better. Materials can be read at home.
Actually, a pretty nice day
I guess it was an exceptional situation today, so I don't think I could give really good feedback
In the notes, the total loss is sometimes averaged? Needs to be made more coherent maybe?
The notes were generally quite good and readable. I suppose I would've liked them to position Section 2.2 and the PBO more clearly as "here are some ad-hoc concessions practitioners make to reality" rather than "here is the correct and mathematically justified fix to a technical problem". Also, I thought unrolling was substantially more interesting than influence functions, so I might've liked to spend more time on it.
I think everything was pretty clear, and we got a good overview from the lectures but perhaps could have gone into more depth into a few things. This was understandably made more difficult by things being cut short.
Scaling limits of Hessian-inverse approximations.
make online lectures possible to listen on people's own device: the lecture area is too crowded and the screen is too small
I would have liked to learn more about the material and was sorry that this material was on a day that's been cut short

Changed my mind3

n/a
This might be the most important topic there is. From unlearning and ablation to narrow misalignment. Albeit it does run a risk of becoming needlessly mathematized, losing connection with reality
Attribution in deep models breaks down exactly at singularities.

Other thoughts3

n/a
Forgot to mention about the previous day with Ashe -- choosing our own agenda was awesome. In fact, it did make me think that Logical Induction might be a promising approach.
Clear, elegant math and great presentation.

Launch Day + proposal round 2 Fri 2 Oct, Leonard Bereska 20 responses · energy 3.8/5

Overall +0.11 as expected n = 18

Content 0.00 as expected n = 18

Clarity +0.17 as expected n = 18

Motivation +0.39 better than expected n = 18

Exercises +1.25 better than expected n = 4

Readings +0.40 better than expected n = 5

Other activities +0.63 better than expected n = 8

Pace +0.44 too fast n = 18

Difficulty +0.28 too hard n = 18

Novelty +0.17 an even mix n = 18

Read what they wrote28

Worked well7

I think I ended up with quite a good proposal :)
Proposal days are great!
Direct feedback from mentors on proposal feasibility.
THe fact that the intermediate submission coincides with the final one. Also, the Launch Day presentation! Very helpful
Getting feedback from other people
Kyriakos (I believe that was his name) is fantastic and really helped me out with his advice on my proposal
Networking with the mentors / fellows.

Would improve11

It's a little difficult to feel like there's enough time to come up with a project proposal, but it's nevertheless a fun and rewarding exercise.
More 1-on-1 mentor matching slots.
I don't know for sure -- but developing ideas for theoretical projects, especially on Agent Foundations, was definitely harder in comparison to the empirical ones. I wonder if it's better to ask people to start thinking about the proposal, and announce that it will share the identical structure to the first one several days ahead.
Not sure proposal days are valuable for me.
I think the same as from previous proposal day
More structure please.
I thought this was great.
I enjoyed today.
Nothing had a great day! Actually, I think it was a bit annoying to work in groups during the last proposal day and then alone today, I felt I would have rather inverted it, or had both days the same, or had the option on both days
In the last project day, we were strongly encouraged to work in group. Today, instead, we were asked to work alone with no further explanation. That felt a bit arbitrary.
I think it is really difficult to come up with a proposal in our situation

Changed my mind6

Reading more about debate, I think it is a very interesting topic and something I want to look into more.
Sharper scoping makes high-risk research questions tractable.
Coming up with a novel proposal is hard when Claude tells you that it has largely been done. Very discouraging. But it is the research reality, and it's good that literature review has become so much more simpler.
n/a
I feel like on the last day I have become very driven to do something, I dont really know what just yet and need to figure that out, but I plan on working on some projects over the next few months
I got some advice from Zach on my future career in AI safety which I considered really helpful

Other thoughts4

Incredible program overall, huge thanks to the team.
I have slipped on feedback forms starting Tuesday this week, and only filled them today. I am wondering how common of a phenomenon that is. Sorry for that nonetheless.
n/a
Please change the feedback form so that it matches a day almost entirely devoted to drafting a project. Thanks!

Reading the bars

Every bar is the answers themselves, laid on one axis with the middle of the scale at the centre line. Left of centre is worse, right of centre better, and the widths are comparable from row to row.

much below below as expected above much above

Pace and difficulty are a direction, not a quality: the centre is the target and both ends are a drift away from it.

on target one step off two steps off

A row says n only where fewer people answered it than answered the day, such as a question about readings on a day that set none.