Aden Power Not you?

By cohort

June 2026

November 2026October 2026September 2026August 2026June 2026April 2026

Rated 1 to 5 in the absolute. Not on an axis with the signed means of later cohorts.

AI Alignment: an introduction Mon 8 Jun, Leon Lang 12 responses · energy 3.4/5
Content 3.7 of 5
Teaching and format 3.4 of 5
Difficulty match 2.5 of 5
Energy 3.4 of 5
Read what they wrote13
In their own words10
  • The general format was good. I found the second topic most interesting and would have liked more time to understand the results related to inductive biases.
  • Different details of alignment were somewhat tricky. I.e. I had trouble wrapping my head around GMG, as well as the motivation behind the training stories. The afternoon sessions were really nice and useful – people were quite engaged. The weather?
  • I enjoyed meeting many interesting people and getting my head int the AI alignment problem space. Points to improve: discussions could be planned better (who reads what, how to divide groups, which reading needs how much time), some texts chosen were low quality or simply too vague to be useful in my opinion; I would appreciate if we try to discuss basic assumptions that go into the arguments stated and prioritize texts that do include rigorous arguments as opposed to story telling.
  • I liked the leisurely pace for a first day, it was engaging without becoming overwhelming. I would like to have a glossary of terms, since it's all pretty new, and more structured exposition. I also would have liked a clearer statement of the context
  • Nice introduction to the main contents of the module. The 1-1's were also very stimulating, although I was not very familiar with some of the topics presented/discussed in the material. Yet, the discussions with the other participants helped me a lot clarify some concepts and technical notions.
  • general overview was good; the slides were clean and well-explained; the discussion groups were nice; the reading was a lot for the amount of time, but overall it still worked quite well. I was able to think this through more carefully
  • Time for reading was a bit short for the lengths of texts. Discussion rounds worked well, maybe brief summaries of what was discussed in each group could be useful to other groups.
  • First day, so mostly looking at readings and posts that most people familiar with AI Safety have already seen so that's why the rating for difficulty. 4 (and not 5) for Content and Format because it took some time to get started today as we didn't have our allocated space ready in the morning, and we shuffled the activities a bit.
  • - The lecture and the readings worked well - The 1-1s were very nice to meet other participants - I think the 1-1s should have had some bell ringing to inform everyone simultaneously that it was time to switch pair. In practice, people went over time and I found it hard to meet many different people. - I liked distributing readings - I think it would have been nice to be forced to change discussion groups more in the afternoon
  • Read-then-discuss small-group format worked well, the discussions were the most valuable part, more than the slides. Most useful for me was Session 2 (alignment-problem decompositions): emergent misalignment is my own research area (I ran a BlueDot EM sprint fine-tuning a 7B), so it was genuinely useful to pressure-test it with people coming from SLT and agent-foundations angles. Session 4's empirical-vs-theoretical framing also lined up well with my AI-control work. To improve: a lot of the Day-1 intro was review for those with prior alignment background (BlueDot or equivalent) - a fast-track option or an optional deeper parallel track on the heavier days would help us spend more time at the frontier.
Other thoughts3
  • Drained because of sleep schedule, the day was well organized.
  • no
  • Enjoying it
Mysteries of Deep Learning Tue 9 Jun, Garrett Baker 13 responses · energy 3.7/5
Content 4.0 of 5
Teaching and format 3.2 of 5
Difficulty match 3.5 of 5
Energy 3.7 of 5
Read what they wrote15
In their own words12
  • I thought the readings were good but I would have liked to have time for more mathematical detail. I think we could have spent more time on reading and less on discussion today. The reason I'd say this about today compared to yesterday is that today was a bit more technical so discussion questions were comprehension questions that could perhaps be answered more quickly. I think we didn't need to spend as much time on the intro lecture in the morning.
  • I really like the discussion format, I feel I learn a lot interacting with others. The questions time after the talk was too long, I would have much more questions after reading the texts. I do not like the military vibe of "you read this now fast". The amount of text we are supposed to read _and understand_ is unreasonable for someone not already familiar with most of the concepts. Perhaps a feature, not a bug, but worth sharing I guess. The questions for discussion sessions are rarely useful or well phrased, sometimes they have litttle to do with the content and ofter they are quiter shallow - this could definitely be improved.
  • Discussion sessions are not so helpful I think because I feel like we are discussing on topics none of us grasped perfectly and there is no one guiding, fixing or explaining things clearly.
  • I felt there was some confusion about the initial lecture. There was _a lot_ of time reserved for questions, but this was unexpected for me. I.e. not announced etc
  • - I was quite happy with the topics covered generally - The presentation in the morning didn't have enough content, or give enough prompts for interesting questions to last the time it did - I feel like the discussions with my group are difficult because we have different mental models of things, but it has the advantage of sharing ideas and learning from each other.
  • I would've appreciated if the lecture was given in person, something similar to frontal lectures with a black/white board. Amongst the different topics covered today, depth separation was probably the most interesting although I'd liked more details on this.
  • Good format: lecture > readings > discussion groups. Only quip B.2 came before B.1, and so some of the necessary background was assumed. But all in all, very nice day. Learnt some interesting material today.
  • The program synthesis framing tied the day together. The most useful part was seeing approximation, generalization, and optimization through that lens, especially why optimization is a separate mystery. The main improvement would be a short primer on the algebraic geometry vocabulary before discussion, since a lot of people got stuck on definitions.
  • Keeping the same groups was a good idea, as we could refer to earlier points of the discussion. Also the bit of feedback provided by Garrett walking by and eavesdropping was helpful without being overbearing. We all have different backgrounds, but I think it would be helpful to have some baseline technical definitions to make sure we are talking about roughly similar things. But I'm biased towards front-loading definitions and putting together the big picture later, which is the opposite of the pedagogical approach you have picked :)
  • Discussion groups worked well. The program synthesis hypothesis was pretty interesting. I would have liked the opportunity to run some illustrative demos or a problem to work on maybe.
  • The talk and discussion period in the morning felt a bit stretched out and could have been compressed slightly. That time was missing later for reading and discussing the papers.
  • During discussions in smaller groups I'm not always 100% sure if we arrive at proper conclusions for the proposed questions. Maybe we should combine these small group discussion with a "general one" involving all the participants?
Other thoughts3
  • I am new to the field so it's quite intense for me but I think I'm doing fine so far, learnig a lot.
  • Not yet
  • Really valued the small-group discussion format.
Singular Learning Theory Wed 10 Jun, Kai Ogden 12 responses · energy 3.8/5
Content 4.3 of 5
Teaching and format 4.3 of 5
Difficulty match 3.3 of 5
Energy 3.8 of 5
Read what they wrote11
In their own words10
  • The lecture notes are great, teachers accessible and explaining very well. I was slightly confused during the lecture but things clarified later during the day.
  • I think it worked well today. We had tight feedback from the teachers and we had timely breaks.
  • Interesting material. The opening talk could be improved, I don't think it conveyed the main ideas very clearly. The discussion leaders (TAs?) were very good! About the format, I felt rushed to red and do exercises at a particular rhythm. I don't necessarily see the point of sitting around reading some notes compiled by people who are in the room and could be exposing the material on a white board. This could also led to more interaction with participants. In general, I have a strong preference for whiteboard/blackboard talks as opposed to slide talks - whenever the formulae allow. I personally would have liked to spend more time on this topic. The idea of the learning coefficient is very interesting, and the expansion of the free energy formula deserved a longer treatment!
  • Content was super interesting and lecturers explained technical concepts well. I think the problems would have been better run in a group work setting (but that could have made the time crunch worse).
  • I very much enjoyed the technical contents of today's module. It felt a little bit rushed due to limited time. Personally, I'd have given more time to main contents of the lectures and spent less time on the introductory part. For the lectures, I must confess that the alternation between discussions and exercises piad off at the end of the day, and it would've been better withouth the rush. I would maybe use more the white/black board for the explanations, rather than displaying slides/documents in the TV (hard to see).
  • - The explanations were crystal clear - The most interesting for me was the connection with safety and the connection with empirical results - I think doing the reading under a constant timer was a bit stressful. It might have been better to have longer (40 min) sections instead of 15 min sections to reduce this constant time-pressure
  • nice to have concrete exercises, but only doing it alone felt more isolating than the previous days of discussions
  • Good format. Having lecturers in person today was great, and felt better than yesterday. Kai and Matthew were both extremely helpful and effectively conveyed their enthusiasm for SLT as a proposal for AI Safety. One criticism perhaps is that we covered quite a lot of material today- we literally tried to complete the full 60 page SLT notes by the authors.
  • I liked getting time to work on problems. I think it might have been helpful if the intro at the start of the day was much quicker, since I, and probably others have more energy early in the day, so that's the best time to do problems. There was a fair bit of discussion in between problem sessions and I think that plus some sort of lecture like Zach's at the end would suffice. Tldr; I think more details first, generalities later could have been a nicer workflow today.
  • Talk pacing and content were good, very well-prepared reading material, exercises varied from very simple to interesting
Other thoughts1
  • there is a typo in ex. 2.13 (b) - "any" appears twice (pg. 22 lecture notes)
Training Dynamics Thu 11 Jun, Guillaume Corlouer 14 responses · energy 3.1/5
Content 4.1 of 5
Teaching and format 3.9 of 5
Difficulty match 3.0 of 5
Energy 3.1 of 5
Read what they wrote10
In their own words9
  • Good presentation and interesting topic Exercises were mostly interesting but some too much plug in and simplify. Time to read paper was too short, so discussion groups did not really form. Maybe from the beginning organize into groups of people who want to read either the same or different papers to then discuss.
  • Excellent choice of material and clear exposition, well-calibrated exercises and interesting papers to read as a follow-up. The time to read the papers and discuss was too short - a lot of people just ended up reading for the full hour +. But I don't have a problem with that. Guillaume has done a nice job!
  • Interesting topic. The content was good but the exercises were even better. I liked that today we got more time than yesterday to really sit down and do problems, and in the process flesh out our understanding. The paper reading part of the day could've been a little more organized but people's energy also starts dwindling by the afternoon (especially given it's thursday!) so the open-endedness fit well. In terms of improvements, before the lecture in the morning, some background reading on the Neural Tangent Kernel could've been helpful.
  • there was too much choice in readings and exercises so it was difficult to discuss later on. a little bit more structure could be helpful, less choice or to agree beforehand that we'll present to each other what we've read once we meet
  • I really liked the exercises, they actually helped me understand the material as I spent time with it. I did not understand much in the lecture except what I had already read. The exercises and the results in them were the most interesting today (only did until question 3, need to do the later ones). On my side, I would improve the preparation I make before the session starts, and on the program's side, I hope some more time is spent with the material. Feels like too much happening in a few hours. Maybe that is the purpose, not sure.
  • - I loved the reading. The papers were very interesting - I feel like we didn't have enough time to read the papers properly. I would have preferred more focus on reading and less on exercises.
  • Having a choice between papers to read was nice! The lecture was great too. I really think, again, that more group work would have made the exercises more productive.
  • Today's content was very informative. The presentation given in the morning session exposed the main ideas and concepts of deep linear networks. The exercises were also helpful and doable. It might be useful to add a hands-on session in the schedule, so that participants can see how the concepts exposed in the talk can be obtained in practice. I'd have appreciated if the tutor was more engaged in the afternoon session, maybe giving inputs to steer the discussion (similar to the SLT module that took place yesterday). Other than that, the suggested material was very interesting and in line with the content of the module.
  • interesting new perspective, the teaching was less structured than usual, was fine, but maybe more structure would be better
Other thoughts1
  • Meta - feedback: maybe instead / in addition to easy/hard you could measure how well-aligned the material was to the participants' background. eg. today I wasn't bored reading the "GEOMETRY OF THE FIBERS" paper, it was a comfortable level of challenge, but it was very aligned with my background, so "easy" in the sense it was familiar. By contrast, some of the reading on the first day was challenging because it was quite far from my background, but the concepts themselves weren't too hard. Maybe it's a distinction without a difference
Decision Theory and Reinforcement Learning Fri 12 Jun, Fernando Rosas 14 responses · energy 3.6/5
Content 4.4 of 5
Teaching and format 4.4 of 5
Difficulty match 2.7 of 5
Energy 3.6 of 5
Read what they wrote13
In their own words12
  • I particularly enjoyed today's problems. I'd seen quite a few of the concepts before but Fernando provided great intuition to solidify them for me.
  • Two lectures worked well for me
  • More time was spent on lecture with slides than was necessary. I think the slides used today could be read independently without making the content less clear. I think doing something like slides and exercises at the same time--similar to what we did the second half of the SLT day, where we'd discuss for like 1-2 min what a section of the notes was about and then spend some time doing problems, and then repeat that cycle at regular intervals would be effective for todays content. I also personally had less interest in utility and RL to begin with, although given that it's included as a topic, I think it would be more fun to address this one from an empirical/engineering standpoint.
  • The more interactive format of the talks worked very well. good reading materials and exercises with good level of difficulty
  • Best part was deriving reward from preferences in one line: preference -> vNM utility -> reward -> MDP -> Bellman/policy iteration. The gamma=1/2 and policy-improvement exercises each carried a real idea. Would tie the end-of-deck agent taxonomy back to alignment more explicitly.
  • The session today was peak. Very good motivation, very good content, felt like I got to learn a lot. As a math and cs major with no physics background though, I found that stuff a little hard and not something I understood (the decomposition theorem and the energy relation with the utility stuff, formally). I will work on that, and if there are any suggestions, please do give them to me.
  • I was not very familiar with reinforcement learning and today's module really helped me grasp the basic concepts underlying this approach. The lecturer/speaker did a great job in conveying the main points of RL and I think I'll follow up to learn more. I also appreciated how the speaker organised the two blocks: the first more first-principles based, setting the context; the second had more practical implications, building upon the foundations of the first block. I wouldn't change much in my opinion.
  • Professor Rosas is an excellent lecturer and I greatly enjoyed today's sessions.
  • having 2 lectures kept me focused and also categorized the topics we covered and we also had a chance of solving exercises related with both sessions and exercises were quite beneficial which enhanced my understanding
  • - I liked the lectures - In particular, the content on the assumptions behind the reward hypothesis was very interesting - Personally, I feel like I was missing some reading to go more in depth - The exercises were useful to consolidate concepts, but the ideas were already quite clear from the lectures, so I would have preferred something to expand conceptual knowledge (e.g. think about RLHF)
  • I liked the afternoon session a little better. I appreciate the idea of trying to motivate (and critique) the standard assumptions of decision theory, but I think the result is a little too meta and confusing. It's hard to explain and critique something simultaneously. I would have appreciated either a little more precision in that first talk, or homogeneous simplifying assumptions. Instead, the notation there and in the notes/problems is inconsistent and confusing.
  • great lecture, especially the first part motivating reward, the second part was more standard RL but still good to have a refresher on that. The exercises are good, but it's hard to do all of them in the time, and I'm wondering if there's a better format (more structured) to do this more didactically, if explicitly splitting in groups or allocating certain time to exercises and certain time to discussions would be helpful. By default, people just do their own thing and that's good, but may be easy to get distracted, especially if there's no (peer) pressure to actually go through the exercises.
Other thoughts1
  • The in-session worked exercises were the highest-value part. More of that.
AIXI Mon 15 Jun, David Quarel 12 responses · energy 3.8/5
Content 3.9 of 5
Teaching and format 4.2 of 5
Difficulty match 3.1 of 5
Energy 3.8 of 5
Read what they wrote12
In their own words9
  • Very good notes and carefully written exercise sheet. It covered some of the gaps (imprecise definitions/ non uniform notation) in the notes and problem set from day 5.
  • The day was structures well and the handouts were nice. David was very energetic and very helpful, and got me interested in the topic.
  • the lecture portion was fast, which I really appreciated! it was nice to spend most of the day just working on problems.
  • the solomonoff to aixi arc was the strongest thread so far: universal prediction, the bayesian mixture, the cumulative bound, and the k(mu) ln 2 complexity bound as a formal occam's razor built cleanly into the universal agent. the worked exercises were the highest value part. the safety caveats, like non exploration, wireheading, and bad priors, made it much more grounded. would improve: a bit more connection between the end of deck agent taxonomy and the alignment motivation, since that is where the clean theory seems to stop.
  • - I didn't like the focus on exercises - I feel like more discussion time, or a more extended lecture would have contributed to getting more take-aways from the day - In particular, I would have liked the connection with safety to be clearer
  • I liked both sessions. The two presentations felt a bit rushed, but they were nicely complemented by the worksheets that summarised the main points and definitions. I still don’t see how this formalisation of general intelligence could help us gain more insight on modern architectures and obtain practical results, but I’ll wait until tomorrow’s session that will be more hands-on.
  • exercises in pairs was fun, the lecture wasn't too long, more time for exercises
  • I really liked the teaching format, but it was a clear case where it's difficult to relate to the utility without some historical context. More of that would be great. Even if it would be of the sort: "Well we wanted to answer such such and such, but that's actually really hard because so and so". It's a rather abstract topic, and I'm not quite convinced that AI will move naturally to a direction where this is a good model.
  • Given David's perspective on the usefulness of this content, I wonder if there would have been a more useful topic to cover. I really did enjoy today though!
Other thoughts3
  • The in-session worked exercises and the live derivations were the best format. More of that.
  • I am personally considering working on AI safety when I finish Iliad, and I suspect other participants are too. I think it would be nice if every teacher highlighted the most promising research agenda/ research opportunities related to the content of the day. I feel like this has been mentioned on multiple days, but it would have been great to devote more focus and more time to it. e.g. give a 15 min explanation of recent work towards this research agenda; instead mentioning it for just 2 min at the end.
  • It was super cool to meet such an inspiring and sharp person as David! :)
Policy Gradients and Goal Misgeneralisation Tue 16 Jun, David Quarel 11 responses · energy 3.7/5
Content 4.3 of 5
Teaching and format 4.5 of 5
Difficulty match 3.0 of 5
Energy 3.7 of 5
Read what they wrote8
In their own words7
  • It was a little hard to understand the architecture of the code of the morning session. The afternoon session was quite straightforward after the lecture, but maybe it would be more instructive to give away a little less. I'm not familiar with pytorch, and didn't have much time to go over the prereqs, so it was a bit of a steep learning curve, but I feel that I still got something out of the day
  • I liked having plenty of time to explore with coding. And seeing the theory instantiated concretely. Overall, I'm sometimes feeling it would be good to somehow recap the previous days in some way, but not sure how to do that effectively.
  • coding in the afternoon was fun. stuff we did in the morning was pretty hard, since i didnt have experience with pytorch.
  • the policy gradient derivation was clear, especially why the environment terms vanish under the gradient. the best part was linking vpg as the optimizer to goal misgeneralisation as what gets optimized. exercises were a bit easy for those with rl background, so an optional harder track would help.
  • - I really loved being able to see the agent behaving in a gridworld - Coding in pairs was great for our mutual understanding - It would have been interesting to dedicate some time to discussing RLHF, I don't know if we will do it some other time
  • Great session, the exercises were very useful for clicking with the theory we reviewed. David was very engaging and keen to guide you through the exercise if you got stuck.
  • David was a great lecturer; very energetic and gave off a good vibe. Would be great to have him again. The material today was more standard RL, so at times felt like _not_ the best use of lecturer/participants' time as we could use this precious time on a more AI Safety relevant topic and keep the standard RL work as a prereq (as policy gradients are a standard topic covered in all RL courses in Uni). Would be fun to have more frontier RL \cap AI Safety type work (or at least get an intro to frontier work). Pair programming was a good format!
Other thoughts1
  • really enjoying the mix of theory days and coding days. the pacing between the two works well.
Alignment in Practice 2 Wed 17 Jun, Garrett Baker 13 responses · energy 3.2/5
Content 3.2 of 5
Teaching and format 3.2 of 5
Difficulty match 2.5 of 5
Energy 3.2 of 5
Read what they wrote12
In their own words11
  • This day would benefit from having good summarizations of papers provided to emphasize the group exercises and balance the individual reading time. I am not very sold on the control agenda, but I guess that now I’m possibly better equipped to argue with objections to my style of AI safety research better.
  • - I felt a bit stressed with so much reading in such a short time, I felt like I couldn't keep up. - I think the builder/breaker exercise was great and very much prompted me to think critically about the alternative experiment - I think the first reading (blogpost) in session 1 was less important. Maybe it could have been replaced with reading a short summary (the ideas were mostly present in the lecture anyway). This would give us more time for reading the paper and breaking/building.
  • I rated today low on some points because I personally didn't find it engaging and it also was not very convinced that control is more impactful than interpretability. If I had to keep the general topic today, I'd hope to remove all the discussion and overview and just look at specific papers and experiments. The first post we read from in the morning felt by far the most substantive to me.
  • Garret seems broadly informed, but wasn't able to be too specific about questions. Furthermore, the slides we're aesthetically very nice, but it was also clearly clauded to a point where one shouldn't actually read the slides. This made it a bit taxing to follow along.
  • the strongest thread was the measurement side: model incrimination, prompt counterfactuals, and decorrelated checks. the scheming vs confusion frame gave me a practical checklist. builder breaker was especially useful because the persona vs goal question is still open. on control, the safety usefulness tradeoff, why trusted editing can beat trusted monitoring, and control as correction rather than full alignment were the clearest ideas. best moment: whether publishing the best red team attack teaches future models to evade it. to improve: discussion groups got noisy without facilitation. a simple template per group, claim, flaw, alternative hypothesis, would help compare papers. track 2 game theory would benefit from one fully worked execute, replace, audit example on the board.
  • The lectures were sort of boring, I really liked the readings and discussions thing. The idea was more like explore little, but make sure it makes a lot of sense, so the teaching was nice there. Time was given when people needed it and were slow reading the material.
  • Garrett did well with making the material more engaging but it was (perhaps by design) a bit too reading heavy and hands-off. I also don't know if we are going to use AI control directly for any other topic so unless the plan is for Iliad to also launch an AI control research team, I'm not sure how today fits in with the theme of the Intensive. This stuff useful to know (and some of us have seen this material in other places like BlueDot courses) but not sure if this is the most valuable day for Iliad Intensives.
  • I think these are key topics to cover. But the slides need work, both in terms of overall structure and in terms of specific content (nonsensical lines). I got almost nothing in terms of overarching conceptual structure, but the "experiments" I read about where interesting.
  • The contents of today's module provided a complementary view to the theory-based approach we've reviewed so far, and this is good. It focused more on prosaic alignment, namely on the role of empirical experiments to investigate the behaviour of LLMs in various test environments. However, I didn't like how the contents were taught. The reading material was too vast for the limited time we had at our disposal, and I had to rush through the papers. As a result, I couldn't really get all the main points. I think this was common amongst participants, because the discussion was not as constructive as it could have been. I would personally prefer fewer readings, allowing more time for discussion. Finally, I feel that the last presentation didn't actually provide a critique of the approach.
  • I liked the presentation and format of the reading and discussion groups. I found the google docs too full and wordy to focus properly on what's the current task (some slop in there, e.g. focus on this and this section when reading..→ maybe rather just say once "skim and extract what is useful for the downstream task". The presentation had also some slop in it. But overall, the design was very beautiful and I appreciate that a lot and the presentation was great!
  • AI slop on slides is not something we should be accepting. While the plan for the day was good, the quality of lectures lower than average so far and again with Garrett, I dislike his order-giving style.
Other thoughts1
  • i have hands on background here: i built a controlarena monitoring setting at an apart x redwood hackathon and have published work on emergent misalignment, so i would value being pushed toward the harder open parts rather than the introductory framing.
Intro to ML Engineering Thu 18 Jun, Julian Schulz 12 responses · energy 3.7/5
Content 4.6 of 5
Teaching and format 4.2 of 5
Difficulty match 3.3 of 5
Energy 3.7 of 5
Read what they wrote12
In their own words11
  • 10/10. No comments.
  • The lectures were very good but I was lacking background setup and experience to engage with the practical exercises. I focused on theory instead, read some more literature on transformers.
  • It was very nice, both the teachers were very helpful. I hope we had one more day of this.
  • What worked well was the abstraction first framing and the immediate actionability. I turned the redundancy condition into an SAE experiment this week and found it predicts cross model feature universality, which made the day especially useful. A short worked example for the condensation correspondence bound would help the second paper land more cleanly.
  • - The hands-on agents exercises in the afternoon were great - I think it would have been useful to have a clearer introduction at the beginning of the afternoon - In the morning I participated in the advanced session, and setting the tools up was very useful - A demo would have been nice, I feel like I only had time to set things up, but not to test them much
  • I think I shouldn't have taken the beginner route in the morning. I was inexperienced with pytorch and i wanted an intro to that but i was familiar with transformer architecture stuff already so didnt need review on that. in the afternoon, I spent all the time on setup, since using an api key for claude code on windows made things really difficult, and the recommended setup was impossible for windows.
  • I would have preferred more time for exercises in the morning session ("beginner room").
  • very nice to be able to have a taste of how easy it can be to set up things on runpod, and using claude code in multiple terminals, I'll definitely get the higher usage limit for Claude now.
  • The morning session on basics of ML architecture was fine. From the afternoon I got that Claude Code could be very useful if one wants to code - I just have no sense for what it was doing.
  • Content was great. It was nice to see many best practices from people doing ML research every day. Some of the setup was much higher friction than I expected, possibly it's great to ask some Iliad colleagues to install and test beforehand.
  • The morning session was great and I learned many new technical tips. The second was also good and I discovered Claude features that I wasn't aware of. However, the teaching format can be improved, maybe starting with some examples demonstrated by the tutor before moving to the exercise of reproducing figures of a paper.
Other thoughts1
  • Strong day. Rare for a maths heavy session to translate into a runnable experiment within the week.
Mechanistic Interpretability Fri 19 Jun, Julian Schulz 10 responses · energy 4.2/5
Content 4.4 of 5
Teaching and format 4.3 of 5
Difficulty match 2.7 of 5
Energy 4.2 of 5
Read what they wrote9
In their own words9
  • The instructor was very energetic, which was pretty nice. I thought too much was covered too fast, and stuff that I did not know was very hard to understand. I don't really get along with long lecture sessions. Also it was very clear that the content that was being covered was not being done rigorously, so there were a lot of gaps. I can particularly say this because I knew some of the stuff already. This is a clear consequence of trying to fit in too much in a few hours. If that was the motive, alright, because I am clearly going to be covering the entire thing now properly. It was very nice to get the exercises as well, I can go through them later to help understand everything. In general, I hope we have interactive notebooks for every session.
  • Very interactive, good exercises
  • Today was my favorite yet!
  • Great high level overview. It covered a lot of ground giving enough detail to be intelligible but without getting too sidetracked by details. Having a lecture interspersed with occasional exercises worked great. I had a moment to gather my thoughts and make sure I was following. Also switching between these 2 modes made me feel less fatigued. Overall, it was a pretty interesting day!
  • Today we reviewed the main concepts of mechanistic interpretability used in AI, and discussed some implications in AI safety and alignment. I found the content very interesting and new to me. I'd maybe try to shrink the discussion and focus on fewer conceptual points, because we didn't have enough time to spend on more complicated methods and approaches. I also enjoyed the exercise, but unfortunately we didn't have time to go through all of them even though they were interesting and could have helped me grasp the theory better. To conclude, a more balanced split between presentation and exercise would improve the module by a lot.
  • Initially it seemed like today was not going to be a very fruitful day till we were doing LogitLens and SAEs because I felt like this is stuff that anyone with interest in Alignment has seen many times over. But after lunch, we did Crosscoders and Transcoders: it was good to get up to speed with the methodological developments of the last 18 months, and get an idea of how the empirical alignment research landscape has progressed, especially NLAs and Activation Oracles. We had the Dan Murfet also today so we didn't get time to do exercises but having time to play around with CLTS, AOs and NLAs would've been helpful.
  • did not attend the first half (know the basics of mechinterp), second half was well-done and I learnt some things.
  • The lectures were fabulously communicated and quite well-designed (aside from some timing issues which caused us to skip a few exercises). I enjoyed the circuit tracing content and have appreciated a more formal approach to it. Dan's talk was cool.
  • I liked the detailed lecture and explanations. Also the topics were in my opinion quite useful and interesting. Also there were exercises which were also helpful.
Computational Mechanics Mon 22 Jun, Xavier Poncini 12 responses · energy 3.8/5
Content 4.8 of 5
Teaching and format 4.5 of 5
Difficulty match 2.8 of 5
Energy 3.8 of 5
Read what they wrote11
In their own words10
  • I am very familiar with the field of comp mech, so this was not new to me. Nevertheless, the lecture was good, the exercises were quite simple even if you are new to the subject. I enjoyed the time in the end trying to build own processes as this tinkering helped understand the relations in intuitive terms.
  • content was very fun!! the paper we read in the afternoon was a super cool result. I wouldn't really change anything about today
  • the arc from hmms to belief states to the mixed state presentation was excellent, ending in transformers encoding belief geometry in the residual stream.
  • I liked the (handwritten) slides. The teaching was good. The exercises were a bit tightly timed, but well selected (maybe with these notebooks it's good to specify precisely what is needed, like "needs an assert". The hints gave away too much, I'd say, but overall I liked the exercises.
  • I've never heard about computational mechanics before, and I found it nice to have a good introduction to topic. The last exercise, where we had to come up with our process, was fun and instructive.
  • Content was well-paced with exercises and just incredibly cool and surprising. The last segment was a great opportunity for playfully handling a bunch of the day's material.
  • - The "Design your own process" exercise was fun and interesting - The coding exercises seemed pointless to me, we were only implementing certain equations, but this didn't really advance our understanding - The explanation of reduction from one HMM to another HMM was very unclear - The notation in the slides was sometimes unclear - I would replace the coding exercises with more conceptual exercises, or at least one of them
  • The lecture was very nice, the instructor was very energetic and knowledgeable, the exercises and discussions were well designed, and overall it was a pretty good day.
  • I liked working on coding examples I also enjoyed the topic it was quite interesting
  • Great exposition and great format, I would have just liked a little more time for the readings, but you can't put everything in 6.5 hours
Other thoughts1
  • the design your own process tutorial with both geometries visualised was the best format, i would welcome more of it.
Alignment in Practice Tue 23 Jun, Evžen Wybitul 14 responses · energy 3.4/5
Content 4.2 of 5
Teaching and format 4.0 of 5
Difficulty match 2.4 of 5
Energy 3.4 of 5
Read what they wrote14
In their own words13
  • I think the debate worked really well.
  • It was quite nice to have a red blue teaming exercise But I felt like we had too many small discussions for very broad topics Topics were quite nice and good to know and I am also planning to read the additional papers mentioned
  • walking the whole pipeline (pre training, post training, deployment) and re deriving each sota method by discussion built real intuition for the affordances each phase gives us.
  • - I loved the blue/red team exercise - I think the content today was very broad, and I really liked that
  • Too much lecture. could make the same points but with less elaboration. The debate session was fun.
  • The empirical approach to safety is interesting and important to learn about, since it is currently the only one that can be implemented and tested in more interesting scenarios. The exposition could be more structured. I did appreciate that it is difficult to organize an interconnected series of topics in a sequential way, and the speaker tried, but I felt that it was mostly cosmetic and more iterations of this lecture could probably help him restructure. I would have appreciated more details on the efficacy of the various techniques and a little more specifics. Maybe something more along the lines of an overview of the literature, with a few deep dives on the actual methodology of one or two select papers.
  • The discussions were well moderated, content super interesting
  • I think this was all great content for folks in the intensive to be aware of, but I'm not so sure how useful it would end up being for someone doing technical alignment research. At the very least, I've developed more vocabulary for reading/discussing general AI safety papers. More exercises or demos would have been appreciated - I felt like I was sitting for a long time during the last block. The content was all well taught and easy to understand.
  • I did not like the structure of the day, and the lectures were very long and boring. I think the topic is a very important topic, but the teaching regarding it was not that great. The discussion part was good.
  • First half of the day was a bit tiresome with the continuous non-stop lectures. Second half was better with the red team-blue team exercise. I feel the materials were interesting but I don't think I have any great takeaways in terms of what I should leave with from the topics that we covered today. Also perhaps it would've been better to have some exercises or tasks where we got our hands dirty with the models, or at least had a walkthrough of a gradient routing implementation or other post/pre-training alignment techniques. Today felt too birds-eye-view/"this is how it works in principle" than it needed to be.
  • By far, the module on practical alignment and safety that I enjoyed the most so far. I really much liked the exposition and I found the blue/red-team session very instructive. I also appreciated that the speaker introduced themselves at the beginning of the session, and I think that the other speakers/lecturers should also comply with this.
  • very well presented, very engaging. too much text on slides, sometimes. discussion groups were very nice, the red-blue team game was good. content was a bit boring/slow at times (maybe because I knew most of it). also could have come earlier in the Iliad
  • The lecture gave a good overview, the discussions were interesting. I liked the red-team blue-team discussions to get an impression on the difficulty of practical AI alignment. Overall, the topic has been noticeably less complex (less formal and mathematical) than other topics.
Other thoughts1
  • the blue red judges design challenge on bioweapon uplift was the best format, i would welcome more adversarial safety case exercises.
Abstractions and Latents Wed 24 Jun, Satya Benson 13 responses · energy 3.0/5
Content 3.1 of 5
Teaching and format 2.8 of 5
Difficulty match 2.9 of 5
Energy 3.0 of 5
Read what they wrote14
In their own words11
  • The first talk felt well-paced, the second talk felt a bit rushed, but the concepts cleared up for me with the successive reading session. The exercises had a good level of difficulty and were insightful to understand the topic. Mandatory homework reading exercises to prepare for the next day after the full intensive day seem a bit much though.
  • I don't think the speakers did a great job in exposing the contets. I found the motivations promising, even though hard to grasp as the speakers couldn't convey them in a clear way. The formalisation of the main concepts were poorly presented and they didn't convince me at all. The exposition lacked mathematical formalism, and many claims were either unprecise or unclear. I also think that some of the concepts presented today are already available in different areas of mathematical statistics–such as information theory and information geometry–from which they could have inherited definitions, notations, etc. Rephrasing the concepts in a standardised framework could help both students (who more likely are familiar with the standard frameworks) and speakers (who don't have to redefine the theory from scratch). Moreover, I believe that this could also help the field build upon settled knowledge, thus facilitating possible advancements. I would have appreciated if, when asked, the speaker had explained possible connections with the notion of the Fisher Information Matrix.
  • The slides were pretty bad, too much text and too small (especially for the condensation part, also speaking too fast for that part). The teaching and exercises were great! I like the website format much better, much easier to navigate than a google doc.
  • Pretty bad lectures. Daniel was able to explain things properly later. It is very important to get the mathematical structure right first, and then it is somewhat conceivable to do some handwaving expecting people to be able to fill those gaps, but at the very start, you need to be very concrete.
  • I have comments for today, from the teaching format to the content being taught: Today stood out as an exceptionally poorly motivated day, even though the general ideas clearly latch onto several lines of research in AI alignment. It was clear that the initial lecture meant to present a rough sketch, which is fine enough. We were presented, however, with some mathematical window dressing, accompanied by technical terms which were in conflict with the story being told. This naturally raises some confusion, which itself was met with downplaying the problems which were obvious to audience. I think this is a problematic way to introduce people to a new subject. Secondly, it's not obvious to me why one would like to spend an entire day on this framework itself. It seems clear that the main points are conceptual and can be quickly taught and elaborated on, whilst the mathematics is mostly vacuous as it stands. This was apparently already obvious many years ago. The theory leaves out many important considerations: How are the spaces for the representations being chosen and constrained? How and why would agents agree on observations? And I'm sure many others I'm not thinking of. And as a postscript: I'm quite stunned by the terminology used in the work. For instance, the word mediator already has a well defined meaning in statistics and causal inference. It's highly suspicious that this word gets hijacked in way that is incompatible with that widely used meaning. These are such adjacent fields, which we should hopefully have some knowledge of.
  • I felt like the introduced concepts were too abstract for me. I think more examples would definitely help with understanding of these ideas.
  • I wasn't able to attend the first half of the day but in the second half we did "Condensation" and finished exercises. The lecture in the second half was okay but not as engaging as other days have been. Abstractions and Latents seems like a fascinating topic and I'm intrigued as to how we'll build on it tomorrow.
  • The first lecture would have been more useful to me if more time was spent to make the relevant mathematical objects clear. I think the general motivation would have been better placed in the reading materials. I thought that the second lecture had too much text on the slides for them to be useful for the audience. I didn't do so many of the problems, but they seemed nice. The condensation paper is too long to read on your own given the time, so a rigorous summary should be made if it's essential content. I did see the blog post summary but found it unsatisfying.
  • - The new Iliad website is nice - The exercises were useful to clarify definitions and work through the concepts - It was nice to have a clear connection with alignment - Discussing in some more depth the different ways in which natural latents/condensation may contribute to safety seems appropriate; we are here to learn about possible contributions to AI safety after all - the morning slides were missing pictures and examples to help us understand the concepts - The example that Leon asked for (two strings of 100 coin flips; the latent is the bias of the coin) was very helpful - A reminder of the definitions of conditional entropy and total correlation; or the requirement for us to review the concepts before today's session would have been useful. I was not familiar with information theory at all and I imagine there were other participants in the same situation. - The concept of abstractions in the morning lecture was unclear to me - The concept of retrieval costs in the afternoon lecture was unclear to me - The afternoon lecture was generally too fast
  • well presented material, the game in the morning worked quite well nice discussions. Time management could be improved: started late, shortened lunch break and then random 25 minutes break, unclear schedule. Also, I think calculating expectation valued of Gaussians on a whiteboard is not a very efficient use of time, while more discussion time could be useful. Going through a couple of round of the game to see if teams can figure out their strategies given more input would be fun. But it was a really good day overall!
  • I don't think this is yet a well-formed theory. I don't think the set-up, the scope, the objects of interest etc have been clearly defined. I am not sure if there is some content in all this that justifies the time investment necessary to acquire this non-standard language. All I was shown are some basic manipulations of the definitions, of the type that follow any mathematical set-up. This is the kind of thing that would be great to discuss when introducing the subject. I also think that ignoring existing statistics and information theory does come with risks. I am not sure that classic work does not cover much of these grounds already, possibly from a slightly different perspective. I am far from an expert, but I see striking similarities between these concepts and main-stream ones. Putting this theory in dialogue with existing ones could allow one, for instance, to import well-known results and guard against known pitfalls. Ideally, this would be explored in a lit review but could not find one in any of the pre-prints and posts. The exposition was poor and confusing and the day unstructured.
Other thoughts3
  • I really do like the course so far.
  • maps straight onto my week's work, redundancy predicts cross-model sae feature transfer, mediation leg is the harder one. more on mediation estimators would help.
  • It would be great to have a copy of the program with the full names of the speakers. It's hard to keep track, especially when people are not introduced and do not introduce themselves.
Agent Foundations Thu 25 Jun, Sung En (Daniel) Chiang 12 responses · energy 3.1/5
Content 3.4 of 5
Teaching and format 2.8 of 5
Difficulty match 3.2 of 5
Energy 3.1 of 5
Read what they wrote13
In their own words11
  • I did't like the topic, found it unreasonable and poorly structured.
  • I think that the slides during lecture had too much text. It was hard to go over all of this content, and at the same time listen to the speaker.
  • - I think I got most out of the lecture - The readings were generally interesting - The cross-pollination prompt for Thermodynamics x Descriptive Agent Foundations wasn't very useful; and we didn't manage to come up with any interesting connections - The descriptive agent foundations ideas were interesting, but some parts like the type signature (A -> B) -> B we didn't really understand very well - Maybe an improvement would be to have the teachers move around and come to every discussion group to answer questions/help spark discussion - Better cross-pollination prompts would also help - The morning lecture was way too fast, specially taking into account that some concepts are quite complicated - Specific examples accompanying the explanations during the morning lecture would have been useful - It would have been useful to split the morning lecture into two pieces with a break in the middle
  • I found the concepts learned today interesting, especially the Gödel-like Löb theorem, while the relevancy of at least some aspects of agent foundations to practical AI safety can sometimes be questioned (it does not seem the frontier labs care about proving their AIs consistent, feels more GOFAI). The lecture this morning was a bit dense, the 2h could have maybe been split up into two sessions. Also, the slides were a bit text-heavy, with full sentences on the slides fighting for attention with the speaker. Reading material and discussions were interesting and insightful.
  • I really liked the content. Agent foundations are a fascinating topic and the reading lists brought together what I imagine to be a very informative subset of the core articles on the subject. I liked that we got time to discuss with people who had done other readings. Maybe the cross pollination prompts could've been more extensive as in our group we found it difficult to pin down the connection between the Thermodynamics x Optimization ideas and Descriptive Agents. Overall I think it was a fairly informative day. One more suggestion is that maybe the lecture in the morning could've been more engaging as after a time it felt like the lecturer was speaking into a void even though as far as I can tell most people were listening attentively (based on how people related things mentioned in the lecture to discussions in the afternoon). I should also mention that while the readings were quite good today, perhaps there was too much material that was kept as pre-reading/prep. Compared to some of the other days (like prosaic alignment or Mech Interp), Agent Foundations is precisely the kind of area where we would have benefited from having more discussions and/or lectures (even short 20 min ones) on some of these highly technical topics.
  • I do think the content has a place, but the point of the course should be to teach and engage. I think these quite subtle ideas should not be done in handwavy machine gun fashion.
  • I liked the notes i read in advance ("optimization and thermodynamics"). I think the lecture in the morning might have been more effective if there was less text in a larger, more readable font on each slide, and the speaker sounded rushed; I think maybe these are both indicators that the depth might have been to ambitious to jam into one lecture. since there was a ton of overlap between the lecture nd readings, I don't think that level of detail was necessary.
  • -I understand that today was supposed to be a survey, but I think it would have been better to really develop a few compelling ideas rather than many diffuse ones. -I think the lecture slides had too much text to be extremely useful for anyone but the presenter. -The extra resources are nice. I think they could be used in a more primary way as they are more formal. -The optimization and thermodynamics stuff is cool! -Also, a general thing I've noticed is that using LessWrong articles seems to undermine the credibility of the content for some participants. This is particularly damning when the content itself is already quite speculative/fuzzy.
  • The slides were too small, too much text, and the presentation wasn't engaging. The content is good, though. The format of breaking out into discussion groups based on the readings is good in principle, but the discipline to actually do the work was somewhat lacking. Overall, was a good day and the readings were well selected.
  • I wasn't able to understand what this was about.
  • Very interesting formulation of the day, and good discussions with everyone about where the field is in the broader context and where AI alignment is moving. Learnt a lot.
Other thoughts2
  • I look forward to the colder temperatures next week as heat-induced sleep deprivation severely limits my capacity atm.
  • descriptive agent foundations / selection theorems connect directly to natural latents, would like more of that thread.
Decision Theory Fri 26 Jun, Sung En (Daniel) Chiang 10 responses · energy 3.6/5
Content 3.0 of 5
Teaching and format 3.1 of 5
Difficulty match 2.8 of 5
Energy 3.6 of 5
Read what they wrote8
In their own words7
  • slides again too small, presentation is ok. the website is very cool.
  • Today was better than yesterday. Felt like some of the feedback had been taken into account. The structure of the second half of the day was also much better than yesterday with the discussion -> lecture -> tournament. The tournament was a fun exercise. One thing to still improve I suppose would be to make the lectures a little bit clearer as in the morning one it felt like there many unresolved doubts and a lot of back and forth without any agreement.
  • Understanding the differences between EDT, CDT and FDT was interesting, but - like the previous days - the topic feels a bit more GOFAI than applicable to currently more plausible paths to superintelligence. UDT and other some concepts appear a bit more esoteric. The game-theoretic part (commitment races etc) appears to be relevant to AI safety only in very specific and constructed scenarios (multiple AGI agents instead of a singularity) which I find interesting thought experiments, but of questionable usefulness.
  • The last exercise was quite nice and engaging The slides were just text and fontsizes were not readable at all I would be happy to learn more about game theory and then continue this would really be nice and interesting also more connected both for the context of today and in general with the other days
  • The lectures in the latter half of the day were better. I think the morning lectures should start with the examples and then develop the decision theories carefully. Clear definitions are important.
  • - There was too much text on the slides, and the text was too small - It would have been better to accompany definitions with examples more directly (e.g. EDT in the morning) - We very easily forgot about taking breaks
  • overall same feedback as other agent foundations today regarding teaching style, but the content was a bit more fun and I like that we get to do this tournament with the bots at the end. In general though, I think it might be nice if we spent fewer days on agent foundations, and more on SLT, mech interp.
Other thoughts1
  • fdt's detect the same function, is exactly my natural latents question, would love that thread in open source games.
Instrumental Convergence Mon 29 Jun, Leon Lang 9 responses · energy 3.3/5
Content 3.8 of 5
Teaching and format 3.6 of 5
Difficulty match 2.8 of 5
Energy 3.3 of 5
Read what they wrote9
In their own words9
  • I found the content interesting! The mathematical formalisms provided a valuable way for me to think about concepts we'd only discussed in an imprecise way so far which was greatly appreciated. That said, I'm not sure I buy that this is quite the right language. It seems to me that any walker on state space with some sort of stochastic element will have a tendency to end up in denser areas. I think that the exercises could be made more challenging. I felt like I could solve many of them just by staring for a couple seconds.
  • - I liked the lecture - As Theo said, it would be nice to have a smoother transition from the exercise sheet to the papers, by introduction the correspondence between concepts in the exercises and concepts in the papers - I missed a more general discussion of arguments for power-seeking behaviour, separate from the idea of retargetability - I think the group discussions lacked good discussion prompts. We spent a lot of time discussing the detailed results of the papers in my group and it would have probably been more productive to discuss the general questions so that we could focus on the ideas behind the papers - The final discussion of limitations didn't work very well. Each objection might have required longer to discuss, and we ended up going on tangents, so we didn't have time to discuss many objections - I think a 3 min rant per person would have been better
  • I found the topic genuinely interesting although I think applications to realistic scenarios are rather limited at the moment. Nonetheless, the assumptions are spelled out in a clear way and the conclusions were accessible through small and motivated conceptual steps. Maybe the talk could benefit from a couple of slides to present some critiques of the topic. Also I'd try to adapt the notation in the slides to that of the paper(s). Overall, good module.
  • The topic was pretty interesting. I did not really feel much for it during the start of the day but it got better over the time and the learning outcomes I realized towards the end were definitely important.
  • It was an interesting topic, and one where a formalized approach has been attempted with some success. So there was something to learn and discuss, which makes it a good choice of topic. The presentation was ok, maybe it could have been a little clearer in its notation and statements. One idea for a discussion could be asking whether the definition of Power they propose is a good one, and how well it captures the idea we have.
  • Slides were good, but the text could have been larger and a bit more concise (which is true of almost any presentation). Exercises would have been better if solutions were separate or as clickable show/hide content (on a website, the agent foundations module had that nice). The readings were good.
  • I think the lecture in the morning could have been much faster if it was more organized--david quarrel's lectures were a great example of what today could be: state definitions, state results quickly and directly without belaboring setup or intuitions, and then set us free to look at notes, exercises, papers in detail. The perspective given in the notes seemed a bit more motivated than in the papers, but it would have been nice if there was some section of the notes that was done in the language and context of the papers to better set up the afternoon readings. more generally, I think the past four days or on agent foundations and adjacent stuff could be condensed a lot. I would have liked have more time to spend on SLT, computational mechanics, and mech interp, since I think there's a lot more detail to be unpacked there, whereas the recent days have just sort of drawn out formalizing philosophical ideas that more or less amount to choosing notation rather than being mathematically involved.
  • The presentation and exercise material were well-prepared and understandable. A bit too much time was allocated for the small-round discussions in the afternoon, I felt we ran out of discussion points at some point. The original papers use a slightly different formalism, and some transition would have helped to more quickly understand their equivalence.
  • author doubts his own proof.
Debate and Provably Safe AI Tue 30 Jun, Stephan Wäldchen 11 responses · energy 3.3/5
Content 4.5 of 5
Teaching and format 4.5 of 5
Difficulty match 2.7 of 5
Energy 3.3 of 5
Read what they wrote11
In their own words10
  • I really enjoyed the part about AIS via Debate, and I found the reading (Debating with More Persuasive...) very interesting. In general, Stephan gave a nice overview of the contents, withtout leaving out critiques and opinions–very honest. I also appreciated the reading list. I think the selected papers were pedagogical and helped me understand the concepts without overcomplicating them too much.
  • I appreciated how a well-posed safety case and empirics clearly accompanied the theoretical work we did. The content was well taught. I particularly enjoyed the exercise and obfuscated arguments sections. I found everything helpful except for the first paper reading + discussion. I don't think the setting needed more clarification. I would have appreciated a more technical reading here.
  • Provably safe AI is an interesting research direction, answering the practical question how we could trust outcomes of non-trustworthy AIs. While the provable framework with a given world model and verifier appears a bit limited in practice, the debate framework appears to me as a good practical approach to work towards AI alignment. The presentations, reading material and exercises worked well for me to get the subject across.
  • Good introduction and motivation
  • - I really loved the safety case. It is the level of detail that I want from a safety argument - I liked the breadth of topics covered - We didn't have any breaks. I would have really liked to have a 10 min break in the morning and another 10 min break in the afternoon - It would have been useful to receive some reminder of the pre-requisites so we could review a bit before the lesson. This was not an issue for me personally, I think I could follow from my previous knowledge of Turing machines, but it is probably useful for others.
  • Today's discussion format was the best in my opinion. In previous days in each group everyone read different papers and we tried to discuss all but today each group members read same paper so we are actually able to discuss and then all groups came together and we learned about each paper also the instructor commented on them which was also quite nice.
  • Very interactive, using the blackboard effectively. A bit more time to prove things on your own could have been useful. the quiz was quite good, the starting with the subtly wrong proofs set the scene very well.
  • I think it's an interesting, realistic research direction. I am not sure I understood the final slide presentation about the safety case. But the formal and semi-formal debate structures are interesting and the empirical results in the paper we read seem at least promising.
  • I found today not as interesting as my favorite topics so far, but the most convincing as an actual step for making AI safe. I thought the format was ok, but I would cut out the readings we did in the morning, since they did not add much to the idea introduced beforehand and they were boring.
  • One of the best days in terms of the format of the day. I think Stephan had scoped the sequence of tasks very well. In the morning, we started with a small lecture followed by paper reading in discussion groups. A discussion group is standard for most days but because we started with it and the time management was good, we got ample time to have all groups present their papers and have a Q&A - something we rarely got time for on other days. Throughout the day he adapted very well to how the energy in the room was, how much time there actually was for each task, and in general he had created a good mix of discussions, lectures and exercises - also keeping in mind the balance between covering the theoretical ideas behind Debate and the experimental studies. His lectures with more blackboard writing than Claude-generated slides were also quite engaging! In terms of improvements, maybe the experimental papers on debate could have had better prompts for discussion but that's a minor thing.
Other thoughts1
  • dug thru the ai debate repo, no stats code or judge verdicts, so numbers are kinda just trust me.
Steganography and Backdoors Wed 1 Jul, Stephan Wäldchen 11 responses · energy 3.3/5
Content 4.5 of 5
Teaching and format 4.3 of 5
Difficulty match 3.0 of 5
Energy 3.3 of 5
Read what they wrote10
In their own words9
  • I think that the game was fun but maybe too long. I would have preferred to spend more time with the theory
  • - The game in the morning was enjoyable and illustrated the difficulties of coming up with a steganographic code - The backdoors topic was fascinating - I liked the dynamic of reading papers in groups and then coming together to discuss the high level ideas of the paper. It allowed us to digest many more take-aways in less time - I missed having a short break in the morning - The break in the afternoon at 14:45 was unnecessarily early in my opinion. Instead of doing two breaks in the afternoon, you can do just one but in the middle of the afternoon (e.g. 16:00-16:15)
  • I found the morning exercise of steganography fun, but taking up a bit too much time maybe. The lectures and discussions were good.
  • The game was a nice intro for the day! The limits of interpretability talk was excellent but a little depressing. I got a lot out of the talks earlier in the day too. It seemed like we were in a bit of a time crunch (perhaps due to our tendency to draw out breaks (: ), so I think cutting one of the reading sections for more problem set time would be a popular decision.
  • Stephan made the first half of the day quite enjoyable, just like yesterday. Louis' lecture was also good but maybe the motivation for the compact proofs research wasn't that clear. A good day again in terms of the mix of lectures, discussions, exercises and the game in the morning.
  • steg game was fun! pretty much all the afternoon content was pretty meaningful. I would not change much today, except, maybe somehow make it a bit more interactive, although idk how this would be done specifically. I liked having the quick problem se session, and if another could somehow be included, I think that would be nice.
  • Good use of table, good explanations, overall time and energy management was quite good. The talk about explainability was nice in that it connected to the methods we did in debate and steganography, but in itself the topic may be not as relevant for AI safety.
  • Good session, nice topics, exercises were instructive. I found the weights extraction via steganography very interesting.
  • I found the initial lecture somewhat boring. The activity (which was designed to be fun), the readings, and the discussion prompts today were well designed and fun. The second half (after lunch) was very good (peak) - the lectures and the exercises were great. I would like to focus on the exercise sheet, which I believe was very well designed and had a very good learning curve. Today was pretty productive.
Other thoughts1
  • last two days(june 30, july 1) were well designed.
Data Attribution Thu 2 Jul, Louis Jaburi 13 responses · energy 3.5/5
Content 4.8 of 5
Teaching and format 4.5 of 5
Difficulty match 2.9 of 5
Energy 3.5 of 5
Read what they wrote11
In their own words10
  • The talks were well-done and the day had a nice narrative structure. The selected papers complemented the day nicely. I found Louis' insights about the field in general as well as his specific research areas particularly useful. I was a bit tired in the second half of the day, but even still, I feel like I didn't have quite enough time with the exercises.
  • Good use of blackboard, calm explanations, very knowledgeable. I especially liked the brainstorming at the beginning and the insistence there to wait until we come up with more interesting ideas. minor errata in the script: defintion → definition typicall → typically
  • Very good lectures, very good structure of the material, and very good actual material such as the curated list of papers, the exercises, and the lecture notes (these were specially very good). In all, one of the best days of the program.
  • I liked the talk and exercises, and the diverse set of papers illustrating different data and safety-related phenomena (deep ignorance, subliminal learning, poisoning, ...) gave a good overview of the problems the field is trying to address. I found the comparison between mech interp as "causal theory of a forward pass" and data attribution as "causal theory of a training run" a nice idea, and was surprised how far this analogy could be taken.
  • The content was really cool to learn about. I was a bit slow at the math so I didn't make much progress on the exercises, and I would have liked a bit more time, which is why i put 4 for the difficulty match. not a huge issue tho since there are solutions at the end which i can reference later if needed. overall seemed like so much to cover in one day, and I think it would be worthwhile to make more time for topics like this.
  • - I loved the morning discussion thinking about what techniques we might use for data attribution
  • I think it was a fairly interesting topic. At times quite technical without really having much space or possibility of making it more intuitive and having (toy) examples. If it can be made a bit more hands on with a practical exercise or demonstration, that would be great but I guess you can't do that with every topic. Louis was a good lecturer and did a good job of breaking down the concepts. I also liked the old-school-blackboard lecture.
  • I find the topic interesting
  • Good discussions at the start of the day which was useful for intuition, but that meant not as much time was spent on the meat of the theory and exercises.
  • I could only make half of the day. In the afternoon the lecture notes were good, the context switching between exercises, lecture and reading was well-paced and it was overall very informative, if a little rushed.
Other thoughts1
  • today's session was good.
Launch Day Fri 3 Jul, Leon Lang 9 responses · energy 3.8/5
Content 4.3 of 5
Teaching and format 4.4 of 5
Difficulty match 2.2 of 5
Energy 3.8 of 5
Read what they wrote7
In their own words6
  • Good way to end the Intensive. Fernando's session was good and David's talk in the morning was laced with all kinds of insider-info which I didn't really know about.
  • The paper about misalignment risk categories was not too convincing to me, I thought it contained too many guessed numbers (how long until research pays off, how quickly would AI take over) and had too strong assumptions (we'll be able to contain slightly super-human AI safely to do research for us) and missed risks other than AI takeover altogether (e.g. gradual disempowerment). The following discussion, however, was interesting and informative - arguably more so than the article itself. I like the goal of ending up with a project idea in general, but the time for finding a project was too short. I think it would have been much better if this goal had been clarified early on, on the first day, so that one has more time to think about different project pitches.
  • thinking about timelines and different plans was very helpful for thinking through my own priorities, making a project plan was extremely valuable, but should have been given more time, maybe even much more time, as in like indicate this throughout the intensive already. similar for the opportunities after, it's unclear how naturally the fellowship follows from the intensive and what information is used for evaluating candidates apart from vibes, could make process a bit more clear (and maybe use for motivation during the course)
  • I think it was a good wrap
  • Nice way to conclude the programme; the session with Fernando was very good and stimulating.
  • - I think it is a very useful exercise to come up with a research idea, even if it is not a well thought out research idea - I would have preferred to be told with some time in advance that today we would present project ideas. If so, I would have spent a bit of time coming up with a project idea. And I thought that would have been very useful. - The discussion today on plans A,B,C,D was useful - I feel like "the usefulness of theoretical/foundational research" has been an elephant in the room from the very beginning, and we should have addressed this before
Other thoughts1
  • thank you for everything. i came in as a builder and now i am leaving with a clear research direction, a real first result, and the confidence to keep going.

Reading the bars

Every bar is the answers themselves, laid on one axis with the middle of the scale at the centre line. Left of centre is worse, right of centre better, and the widths are comparable from row to row.

1 2 3 4 5, best

A row says n only where fewer people answered it than answered the day, such as a question about readings on a day that set none.