Adam Newgas Not you?

By cohort

October 2026

November 2026October 2026September 2026August 2026June 2026April 2026

Rated against what each person expected. 0 is as expected.

Intro to ML Engineering and Transformers Mon 5 Oct, Dan Wilhelm 42 responses · energy 3.7/5
Overall -0.05 as expected
Content +0.26 better than expected
Clarity +0.12 as expected
Motivation +0.38 better than expected
Exercises -0.02 as expected
Readings +0.04 as expected n = 23
Other activities +0.32 better than expected n = 34
Pace -0.31 too slow
Difficulty -0.52 too easy
Novelty -0.60 mostly review
Read what they wrote75
Worked well21
  • The live code examples were nice however they were hard to follow. I wish the code was accessible before the lecture.
  • The discussion was stripped down to the simple essentials, AND it involved live implementation. Both these contributed to efficient learning.
  • Long form disucssions.
  • Exercises and group work.
  • I thought the ability to ask questions mid talk was very useful for clarification.
  • the worked code examples
  • very happy to hear about the various opportunities (but i felt it was out of place at the beginning of "Intro to ML" class day)
  • The discussion during breaks.
  • I knew most of the material covered. Really enjoyed having the opportunity to discuss it with others who felt it was new. I also enjoyed watching the teacher code on the fly in front of others. That is super hard and I was impressed. (Helped me see what kind of skills an AI safety researcher should have).
  • The schedule structure is good.
  • The lecturer was very knowledgeable and answered all the questions well.
  • A very soft landing toward the basic but useful ML concepts was very helpful. The lecture was well organized and well explained. His answers to participants' questions from his own practical experience were very helpful to understand the general idea of the concepts. All the questions from the group were a good addition to the lecture.
  • mixing slides and theoretical explainations with exercises and implementations
  • its a good idea to give an introduction
  • Examples with TransformerLens and GPT-2
  • I loved the attention section, and I liked how easy it was to discuss questions during the lesson.
  • The lecturer is good and his answers were clear
  • I think the engagement with questions was nice.
  • Basic understanding of the transformer model structure
  • People were kind and receptive to chatting/working/eating together.
  • The exercise sessions were useful.
Would improve31
  • I think more breaks would be good. The second session was a marathon.
  • slightly more mathematical / formal clarity
  • The ordering of some of the concepts could be shuffled around to flow better for a math / physics audience.
  • More figures, and perhaps a better feel for the theory narrative: like constantly returning to central intuition for LLMs as a sequence predictor.
  • Some of the attention material was simply presented but not well motivated (in terms of where these blocks came from, why we made certain design decisions).
  • The explanation of the transformer architecture was a bit difficult to follow for me - structures sometimes seemed haphazard/unmotivated, and I got lost in understanding the various different parameters (especially when starting the coding exercises). But, it was not that the material was difficult per se.
  • Bit of a slow start
  • Some more diagrams and structure would be nice. Maybe a general overview of the transformer architechture to start. It may also be worth it to spend more time on the complexities of the transformers etc than on the very basic concepts at the beginning. Although there were lots of questions which makes it hard.
  • Maybe the presentation preparation could be better. (I found 3b1b explanation of attention much more intuitive -- but maybe he oversimplifies.)
  • it was not clear at all we would be using Jupyter notebooks. I specifically did not set my environment up bc the prereqs for this day were just "foundations" the material was very (too?) basic, bit the presentation was confused and unclear. the material should be presented more systematically by, for example, giving an overview of the architecture before explaining each part, defining the variables and where they live, given the intended academic audience.
  • Sorry for late feedback -- felt like some concepts could have been introduced more rigorously, given almost everyone is postgraduate level in maths, physics or CS. Useful review overall.
  • More breaks probably? I think also sectioning the talk into different parts would help better. So there's more granularity on what we're talking about.
  • There were many "why that choice" questions (ie, "please motivate that choice"), which I feel could have been better prepared in advance, knowing the sensibilities of a math-y group of participants. The lecture felt redundant with the 3B1B video series, or other tutorials. It may have been a good use of the day for the participants to be expected to be familiar with it in advance, and work in groups for discussions about the content, to solve various exercises, with some Colab guidance.
  • I think to get the point of residual stream across it may be helpful to show a counter example. Basically residual streams are added to by the (computations done by the) MLPs, attention heads etc. The counter example (what a residual is not) would be to show that the readout of one is the input of the other. I think this might help motivate the idea of the residual stream--why it is a "stream" where stuff gets taken out and added back in.
  • Not much. Since it was the first day, it wasn't technically very new, but I think that is expected.
  • Maybe It would be nice (I don't know if it will be done int he following days) to have more mathematical definitions of the objects we are working it (as example, in the transformers part, as generating kernel), in order to figure out what possible research directions would be
  • Better clarity for the slides and explanations. There are some typos and confusing explanations during the lectures.
  • I think the presentation could be improved. In particular, just practicing giving the lecture before and doing it less on the fly would allow us to spend more time on learning. I think visualizations would do great for learning how attention works, how the numbers flow through the vector, instead of solely relying on the mathematical equations. Perhaps even some interactive thing generated by claude code or something.
  • I could follow the discussion and it was well done. For some other participants maybe they could watch 3Blue1Brown videos.
  • I wish the speaker had rehearsed the presentation, or otherwise prepared it better. Though the speaker clearly knew what they were talking about in the big picture, he was often lost on details he "hadn't reviewed recently". Separately, I wish the presentation was presented with an eye to a mathematical perspective: the audience is mathematicians and theoretical physicists.
  • I would try make the room less stuffy. Maybe AC would help.
  • it was a little slow at the beginning and a little fast in the end. meaning I would have spent more time on the more difficult concepts and less on the easier.
  • the beginning on the "basics" of ML was a bit loose/not as clear the connection to more modern methods and some small mistakes, the attention part was pretty great but the programming problems are mostly bottlenecked by not rembering the syntax rather than not understanding the topic well enough, especially since the formulas are written on the sheet
  • I would prefer the coding challenges to be more difficult. Today they involved writing down math formulas in pytorch syntax, which is okay but does not teach me much.
  • More breaks. I found it hard to focus with such a long session.
  • The introduction could have been more energetic and engaging.
  • Some references for mathematical and computational precision.
  • Some typos, nothing serious.
  • Make the lectures slightly faster
  • I think more examples would be good.
  • Didn’t have much expectations for an intro/refresher day but always happy to recall stuff!
Changed my mind19
  • No changes.
  • not really, but we were only covering an intro to ML
  • Just how much LLM building is trial and error
  • considering applying to this funding! meta: is this the level of preparedness / knowledge of successful researchers / funders?
  • I had a time to properly think about transformer architecture, which was useful.
  • Seeing Dan work in Colab was instructive!
  • I had a really interesting conversation at the beginning of the ice-breaker sessions. I learned some ideas about decision theory that I had questions about. It also opened up new questions that I plan to follow up on.
  • Didn't change my mind on anything today, just a good review on transformers.
  • Before the program started, I was worried about the level of mathematical concepts. This first day provided a good overview of the practical and theoretical understanding of AI/ML models. I got much more comfortable participating into the program.
  • I was surprised at just how simple the recipe for a transformer is.
  • On prospects of procuring required resources from notoriously obscure sources.
  • today was mostly notions, nothing really worth mentioning. I definitely learned a lot!
  • not much this was more introduction
  • About possibly applying for a founder's programme.
  • No
  • n/a
  • No change.
  • We could access the modalities(parameters) of any model
  • I feel more confident in my ability to pivot to AI safety given how much more at place I feel here than expected.
Other thoughts4
  • if the material were less familiar, I would expect to have gotten less out of it, as some of the presentation was not (sufficiently) self-contained
  • Thanks for bringing us together.
  • Nope
  • no
AI Alignment Introduction Tue 6 Oct, Margot Stakenborg 37 responses · energy 4.0/5
Overall +0.57 better than expected
Content +0.54 better than expected
Clarity +0.57 better than expected
Motivation +0.59 better than expected
Exercises -0.20 as expected n = 25
Readings +0.22 as expected
Other activities +0.54 better than expected n = 35
Pace -0.11 about right
Difficulty -0.30 too easy
Novelty +0.11 an even mix
Read what they wrote59
Worked well21
  • margot is an excellent speaker and the ideas were well communicated
  • The presentation and discussion covered a wide variety of topics.
  • The discussions were very fun!
  • The discussion sessions.
  • The discussions about the readings were edifying. I learned a lot about the history of thought in the AI safety field
  • The group discussions after each reading
  • The discussions
  • Discussion periods and socratic method.
  • I thought the conversations were thought provoking. I got a good idea of the various philosophies around alignment and the open questions people generally struggle with.
  • Margot teached super well!
  • Margot's design and delivery and facilitation of discussion was excellent.
  • Lot's of discussions
  • The readings given are nice and we got to discuss them
  • extremely interesting topics with great explainations.
  • time splitting - the decomposition into various parts such as lecture, reading, discussion was well proportioned
  • Really excellent and enthusiastic instruction. Margot's depth of knowledge and ability to communicate are inspirational. I am meaningfully changed for having attended yesterday.
  • Peer discussion
  • reading and discussion sessions were good. It gave to chance to read at our on pace about things that interest us. Discussions gave an overall impression of all other topics.
  • For this flavor of non-technical content, the alternating reading/small-group/whole-group discussion format was great!
  • Margot's lectures were very nice and allow us to gain a clearer picture of the field and a better understanding of the jargon employed in AI safety. It was very nice to go through the details of the Hugging Face incident. Multi-agent interactions played a crucial role in this context, and this motivated me to develop a theoretical framework to investigate interactions between different agents using Partial Information Decomposition. I already had some ideas about this before the intensive, but this lecture helped me arrive at a clearer picture.
  • passionate, clear speaking, extremely knowledgable and eloquent speaker!!
Would improve20
  • A bit more silence during the reading sessions to increase concentration; the amount of reading was too much, and the continuous talking during the reading sessions was not helpful. Also, the lecturer was highly knowledgeable but it would be better if she just spoke a little slower and took some time to breath :)
  • i liked the quiz maybe we can more/harder quizzes that force us to solve some small hypothetical problems
  • Today was a great day! One thing that was a bit tricky for today was that we were asked to do a bunch of reading, but quickly the instructor would engage in interesting but possibly slightly off topic conversations which made it a bit difficult to concentrate on reading.
  • A lot of the discussion veered quite far from the articles in the slides. Unsure if this was intended but a bit more organization would have been helpful.
  • The discussion needs to be more structured, if we have 25 minutes to go through some posts and discuss them in groups then maybe we ask all groups to post a brief summary 2-5 lines on a forum which is a link everyone else can access. That way so instead of asking each group to speak out what they discussed turn by turn, we just take 5 more minutes to read the forum. Time management seems to be something that can be worked on the above is just one suggestion in this vein.
  • Was very good, maybe some more structured question sections?
  • The questions after these do not ask what my original expectations are and thus contain no information about the quality of the presentation
  • A lot of the reading was hard to parse and used a lot of unnecessary jargon
  • Maybe having less time to read the paper and more time to discuss.
  • More concrete examples and connection to the underlying theory of LLMs, perhaps a step-by-step walkthrough of how particular cases of misalignment might work given what's been covered in the previous intro to ML Engineering class.
  • Reading structure
  • I really enjoyed all the discussions. From what I understand the course will not include that much more of this broad overview of alignment philosophy out there and I would just like to hear more about this.
  • I'd go more into detail on the readings before getting us to read them. Or at least, I feel like today felt quit slow/uneventful as a result of so many 45min reading breaks we had.
  • Encourage the readings more. For instance, encourage people to go to a quiet place to read, and take discussions to another room.
  • More concrete examples of alignments vs misalignments. Motivate with important examples/case studies before introducing them.
  • this was a great session, but I would have liked to do it after having acquired more technical knowledge on the models and the training process.
  • It was great a overview, slightly more quantification would have been nice, e.g., real data to and consequently discuss texts like AI 2027 and Europe 2031 from another critical angle
  • More time for readings
  • It is a bit difficult for me to read papers live and then try to comment on them. With only 25 minutes to spend on each of them, I can never get too far into them.
  • this was amazing! i appreciated the reading+discussing format, found it quite distracting that the speaker would answer attendee questions during the reading time! now i need to chose between reading the material and listening to the very knowledgable clear and passionate speaker. and if i try to read, i am constantly distracted
Changed my mind13
  • I feel a bit sorry for the security people at OpenAI after reading the long Tweet.
  • yes i realized why AI safety research is important and how sometimes you can take a step back from doing math and think about why are we doing this
  • Yes, but the exact topics elude me at the moment.
  • More concerned about ai risk
  • I didn’t change my mind on any subjects today
  • Started having an interest in the economics of AI agents
  • Not really.
  • It does seem like there are predictions that the rationalists (pre-LLM) got wrong. I am interested to hear more about this and will follow up on it.
  • I would have probably kept this section for next week or the week after
  • The possibilities of large AI models misaligning is very high
  • I am slightly more sympathetic to controlling AI and/or slowing its progress than before.
  • I was not aware of gradual disempowerment until today, which very pointedly describes a very reasonable concern one could have even in the more conservative cases of capability growth.
  • i found
Other thoughts5
  • N/a
  • Nothing
  • I did not like "from each table a person speaks vaguely about a vague preceding discussion" format. The readings were not matched against the allotted time. I barely got to the point of long papers I had 25 minutes to read -- and then there was no concrete discussion of that point, so I had to talk to ChatGPT to get at least some well-formed takeaway.
  • Nothing bad about the lecture, it was just the content was too philosophical to follow for me personally especially when I was suffering from the jet lag.
  • can we keep going until 6pm on future days?
Mechanistic Interpretability Wed 7 Oct, Dan Wilhelm 38 responses · energy 3.9/5
Overall +0.45 better than expected
Content +0.66 better than expected
Clarity -0.08 as expected
Motivation +0.61 better than expected
Exercises +0.03 as expected n = 35
Readings +0.15 as expected n = 20
Other activities +0.40 better than expected n = 25
Pace +0.74 too fast
Difficulty +0.50 too hard
Novelty +0.79 mostly new
Read what they wrote64
Worked well20
  • The second instructor was very well versed in the material and engaged with everyones questions.
  • very interactive. Participants actively engaged with lecturer by asking good questions and we discussed freely.
  • The overview was broad and looked comprehensive.
  • I liked the coding exercises to reinforce the concepts we learned
  • Exercises
  • The brief presentation given by one of the students. Involvement is nice and sharpens the attention.
  • I think the afternoon lesson was quite nice -- due to examples.
  • The first half was well taught and quite clear.
  • Helped me get an initial but a very useful broad overview of different mech interpret techniques. Starting with CNN helped understand the history of such methods. Later, although there were lots of different ideas presented I understood and am taking home some useful stuffs.
  • I think the exercise where we had to deduce the type of circuit from the attention pattern worked quite well to help me internalize the method.
  • The afternoon session was excellent: good pace, good explanation, good examples & slides.
  • Lots of discussion
  • Julian Schulz' presentation was _amazing_. clarity, enthusiasm, knowledge, humour. overall great. the exercise with the K- and Q-composition was brutal (especially because the first day's on transformers was not clear enough to learn the architecture–i'm glad i spent the evenings learning abt it) but enlightening
  • The circuits expansion was particularly good.
  • Discussion was great during the lectures, i wish we had had more time.
  • I really enjoyed the balance between Dan's transparency and pragmatism on the basics of mech interp and Julian's enthusiasm on the newer methods/approaches.
  • I particularly found the live coding demo useful for the activation diffing.
  • breadth of the material was good, but too much to follow, like i should know all of this but maybe in more depth but I understand its the intensive
  • Discussion was helpful for having people break down concepts and ask questions rapidly
  • I thought the questions and answers were v helpful
Would improve25
  • The pace was faster in the afternoon than in the morning. This might have been due to the teachers' individual preferences. However, I would suggest maintaining a more consistent pace throughout the day.
  • I would add more specific examples into these slides--I feel like I best understand what's going on when there's specific examples. (This is in part because of my background: I am a mathematician and feel like I understand the ML setup less well than the theoretical one. This might not be true for everyone.)
  • Talk slower
  • The pace was quite fast in both halves. Splitting this or pre-readings would have been much more helpful.
  • The speaker talked too fast and the explanations were confusing. I had to read the cited papers to understand what was going on.
  • More structure, fewer slides. (E.g. slides don't even define the J-space. It was defined verbally in 5 seconds, and, as I understood after asking an AI -- not even correctly.) I think if we do exercises at all, we could discuss the details of these python libraries. Also too many vague questions from the audience, very hard to keep the focus
  • A little bit more discussion and hands-on, and more intuitive examples (metaphors) to help understand the complex new concepts.
  • I would increase information density
  • The part on CLTs was difficult to follow for me.
  • Explanations could have been a bit more clear.
  • very quick explainations of many difficult concepts
  • Great! Honestly a lot of information, maybe a bit slower through the more recent mech interp stuff.
  • I really enjoyed today. We did take a lot of time to go through older methods and I would've liked to spend more time going through more recent stuff, especially J-space. That being said, I had pretty decent exposure to previous interp research before NLAs and J-space.
  • More time for the Mech Interp session, part 2. That, or references to expository materials.
  • Was excellent
  • Better explanation of what circuits are. May be give more concrete examples.
  • again, i found the Dan Wilhelm's presentation extremely underwhelming, way below the level or rigour or technical knowledge needed for this audience
  • The slides could be more precise.
  • I wish the style of the presentation was more mathematical, even if that would require a little more notational baggage. (e.g. https://transformer-circuits.pub/2021/framework/index.html )
  • We probably needed the full day to get through the material, so more time would be great. Questions and discussion were helpful.
  • Not much, really enjoyed today!
  • maybe a little more time for exercises, but there was a lot of material so I guess it is hard to fit everything in.
  • less material more in depth
  • Less time on introducing Mech Interp, more time on deep-dive and math/matrix multiplications behind concepts introduced
  • More equations. At least for me, I find the graphs by themselves hard to interpret without the relevant maths alongside.
Changed my mind16
  • Random trial ams error seems much more effective than I would have expected.
  • I did not mech interp was used for safety and alignment.
  • I thought mech interp was doing pretty poorly but it seems like Jacobian lenses are worth more of my time.
  • N/A\
  • J-Space is interesting!
  • Not in particular.
  • Circuits and NLAs are, perhaps, more impactful than I had thought.
  • Mech interp is cool
  • not really, i guess i was surprised to hear that the more "mechanistic" technique of mech-interp are
  • `The mathematics behind mechanistic interpretability is close to non-existence.
  • Yeah, I now find these methods more interesting.
  • Feel more comfortable discussing research and collaborating w/ people here in the near future Feel more motivated to start diving deeper into mech interp
  • not really, but developed my understanding of circuits and cross encoders
  • the NLAs were really cool, the fact that LLMs had hidden logics that NLAs can detect was pretty cool and new
  • I learned about J-space for the first time today and I will definitely look into it more. I also feel that my impression that SAEs were a dead end due to the Deepmind deprioritization was overblown.
  • I didnt realise how quickly mech interp was changing as a field
Other thoughts3
  • N/A
  • Bit more of exercise, like doing coding would have been better. While coding, we could understand better.
  • n/a
Alignment in Practice Thu 8 Oct, Margot Stakenborg 32 responses · energy 3.9/5
Overall +0.57 better than expected n = 30
Content +0.53 better than expected n = 30
Clarity +0.57 better than expected n = 30
Motivation +0.67 better than expected n = 30
Exercises +0.78 better than expected n = 23
Readings +0.46 better than expected n = 28
Other activities +0.76 better than expected n = 29
Pace +0.03 about right n = 30
Difficulty -0.17 about right n = 30
Novelty +0.50 mostly new n = 30
Read what they wrote38
Worked well12
  • I liked the debate in the end :) Good way to help memorise the various topics.
  • the structure, well partitioned between lecture, reading, discussion
  • The instructor-lead parts were focused and clear.
  • The blue/red team exercise at the end was very helpful to make concrete the techniques we read about
  • The final game we did with Margot was very nice. It helped me touching by hand the challenges of making a model safe and preventing from attacks.
  • great resources and amazing, enthusiastic, enthusiastic, knowledgable, clear, and available speaker
  • The red/blue team exercise was extremely helpful for consolidating concepts introduced today
  • Margot's delivery, selection of material, and engagement with questions, is excellent. The discussions were helpful. The debate was enlightening, and clarified some of the key points of the day.
  • The exercise at the end
  • The exercise at the end
  • The exercise was a lot of fun.
  • I liked the activity at the end. It helped me learn at a deeper level.
Would improve15
  • I found the debate enjoyable and interactive.
  • Fewer topics but a selection of modern/popular topics in more detail.
  • It was great a overview, slightly more quantification would have been nice, e.g., real data to and consequently discuss texts like AI 2027 and Europe 2031 from another critical angle
  • For this track, if the goal is to expose those new to the field to the fields' essentials, I'd cover more material during the instruction phases and leave less for discussions.
  • The slides could have used more visuals maybe? I think today was very good
  • the debate was a bit of a miss for me, as no one was cyber-expert so we had no real basis to have an informed discussion / debate. i would have rather spent more time listening to speaker or simply
  • The discussions are good, but could be a bit heuristic and a more concrete code-based tutorial would be nice to better understand alignment in practice
  • A bit of overview of the papers/readings before unstructured reading time to make sure we hit on key concepts
  • nothing
  • SGTM is awesome! Some more time on it -- the mathematics -- could be helpful.
  • Nothing
  • This was a great day: I genuinely don't think there was much to improve. Perhaps during the example of redteaming/blueteaming to not give specific examples and instead let us figure it out on our own.
  • Maybe spend more time on SGTM and gradient routing.
  • Lectures were very professional. I would have liked more time for the game at the end, but I don't know if this is even feasible. My low energy on the day had nothing to do with the lectures themselves.
  • I think the slides could have been more detailed. The reading was a lot more informative than the presentations.
Changed my mind10
  • I did not realise that most of the effort of technical alignment does not go to pre-training.
  • The day has spiked my interest in the red-teaming approach to safety.
  • I didn't
  • I'm much more aware of how even policy and governance decisions can have a technical and research aspect to them like tradeoffs of pacing AI development and deployment
  • Jailbreaking turned out to be even easier than I expected.
  • The safety of open weight models
  • Safety for open weight models seems almost impossible
  • Alignment in practice seems like a Sisyphean enterprise, and I think that it would be worthwhile knowing the tradeoff landscape more quantitatively.
  • The fact that some of these techniques work, tells me a lot about the structure of LLMs that I didnt know before. It really helped to get more perspective/clues on how to think about the model internals.
  • I learned about personas and AI as a way for aligning
Other thoughts1
  • Generally, given the mathematical expertise of the audience, erring on presenting mathematically complete material (when appropriate/available), would be welcome and, I think, efficient.
Decision Theory and RL Fri 9 Oct, David Quarel 42 responses · energy 4.0/5
Overall +0.46 better than expected n = 41
Content +0.51 better than expected n = 41
Clarity +0.32 better than expected n = 41
Motivation +0.12 as expected n = 41
Exercises +0.37 better than expected n = 41
Readings +0.29 better than expected n = 35
Other activities +0.06 as expected n = 32
Pace +0.54 too fast n = 41
Difficulty +0.22 about right n = 41
Novelty +0.66 mostly new n = 41
Read what they wrote55
Worked well17
  • The answers ti questions by the lecturer.
  • The notes introducing Solomonoff induction and AIXI were very well-written and good for internalizing the more difficult conceptual material
  • The Notebook in Colab. And the sarcastic approach of the teacher also worked well, although it sometimes made him slightly difficult to approach (somewhat afraid to ask stupid questions).
  • Got my first exposure to RL through the well designed Notebook.
  • I think the material and discussions with the lecturer were very useful.
  • The exercises were so well curated and detailed.
  • Loud voice, good preparation
  • The material touched on the core concepts of RL that are relevant to understand what is still being used/practiced today. So, after getting the basics I hope to read more and familiarize with the concepts more. Good foundation.
  • Building around the central objects of expected value.
  • The mathematically rigorous presentation helped me, and several of my colleagues, latch onto the ideas and formatively develop intuition. Most of the exercises were well-considered and helpful for advancing intuition.
  • interesting material, amazing energy and knowledge from the speaker
  • The topic and formalisms were new and different from those covered on previous days. I really enjoyed it, and even when the material was difficult, David was very good at explaining it again and answering questions.
  • I liked the rigour and the teaching style. Often was too fast, but having time to go through the material helped, and I believe with this much content you cannot go any slower. So no complaints.
  • The lectures were the best part, but I wish they'd been slower.
  • The materials is well organized and great for me.
  • mix of examples and theory and coding
  • The derivation exercises were helpful.
Would improve27
  • I think the most helpful improvement for both learning about AIXI and Reinforcement learning is: immediately after giving the definition of either of these, give a specific example. For example, it wasn't clear that, when talking about reinforcement learning, that the action one can take only has probabilistic distribution on what actually happens after, and I think this would have been cleared up if the two examples were moved first. Also, it seemed like these slides were originally made when reinforcement learning and AIXI were covered for a longer time; it might be good for the slides to be adjusted rather than to try to cover as much as possible.
  • Great
  • I liked it
  • Reduce the amount of material
  • Slower pace
  • More time explaining relevance of Solomonoff and AIXI results, especially in practice. Less time on mechanical exercises
  • Begin the day with the notebook exercises to motivate the mathematical content. Next, I would reduce the volume of the material covered, and focus on getting the main point across, rather than rushing through multiple large topics in a single day. Of course, this is all specific to my background, and probably doesn't make sense for everyone.
  • I think a lot of stuff was poorly motivated. I wish we would have talked about what was mentioned in the calendar.
  • I think today was great.
  • It was a bit much different stuff crammed into one day, maybe remove the last part, and focus on AIXI so one has more time to read through the later parts
  • Went a bit fast but I honestly given how short the program is this is better than prioritizing real-time understanding. I think I like the speed-run style much better as long as we're given the foundational exercises to work on in our own time to cover our bases.
  • It was great! Although I don't know how much of the first part of section D we really studied (which was interesting to me i.e. the more economics of rationality section)
  • Encourage the audience to filter the questions.
  • Too many materials covered in a short time. The math materials could be covered at a high level (focusing on intuition and proof sketches) instead of going too deep. The colab tutorial at the end is nice and deserves more time.
  • Make the instruction a bit slower.
  • I would welcome more time on this material could -- it seemed interesting, but rushed and cool stuff skipped. Introducing the example from the very end at the beginning could help ground the abstraction through the day.
  • maybe a bit fast at points
  • It felt a bit rushed, but it was okay as I knew most of it beforehand.
  • today was way too packed, with the subjects seemingly weakly related to each other.
  • It was really good and interesting content, and great teacher with great understanding of the content / field. Only issue on my side maybe was that the content links are sometimes a bit messy, as in some links in intensive-october.pages.dev, some in iliad-intensive.org, some in discord. It gets hard to follow. Would be nice to have a simple and compact way to get to all lecture resources. Applies to previous days as well.
  • I could not download one of the packages for the Google Colab.
  • This was probably too much material to fit in one day. I would have preferred to go through some of the theorems slowly in class, rather than going through them on my own afterwards. The notes had a lot of notation, which made it difficult to keep track. Lecturer was clearly very knowledgeable and good at presenting, but it was impossible to get through all the material comfortably within the time frame. The parts that I could follow were great.
  • I feel like, at times, I was struggling to to see the connection bewteen the specifics of decision theory, AIXI, and how it relates to AI safety and existential risk. One question I had going into this particular section was learning what the load bearing assumptions (inside of decision theory) are for existential risk due to AI. I do think I was told these assumptions... I just am not sure what they were. I mean, I feel like I got a sense that the needle is in the haystack of info today but I'm not sure where though. I am excited to look back at the info and learn from it.
  • nothing
  • This is best lecture day this week
  • Very good - personally, I would remove some of the quantity / amount of detail to be tracked. Not in a manner to dilute any one individual topic, but in a manner that - in principle - the material could have been presented at a black board - stripped down to its essentials. I emphasize that this is a personal take, constituting an improvement on an already good program.
  • The pacing was a bit fast, but perhpas the only way to fix that would be to assign pre readings or split this across 2 days.
Changed my mind9
  • I'm better able to understand how to study and bound AI behavior in general rather than just specific models
  • I wasn't aware that there is such deep theory behind RL
  • N/A
  • The importance of RL in the AI world.
  • AIXI seems very interesting!
  • Kolmogorov complexity is a very cool concept and seeing its use to formalize the often-cited 'occam's razor' concept was very interesting.
  • I am very interested in the AIXI/Solomonoff Induction research agenda.
  • I didn't think this topic was important/interesting before, and my mind has been changed.
  • I do think that I underestimated how much the AI community understood mis-generalization and I am curious to learn more about how they think about it.
Other thoughts2
  • N/A
  • The material was bit too dense for me. I knew about RL before in different context, not the theoretical basis of it. So, having everything on the same day felt like a lot. The theory session itself could have been a full day. The coding session, where we were given a code to play with was too short to really understand what is happening. Spending more time on that would have been more useful for someone like me who has more coding background than the math.
Steganography and Backdoors Mon 12 Oct, Julian Schulz 33 responses · energy 4.2/5
Overall +0.22 as expected n = 32
Content +0.25 better than expected n = 32
Clarity -0.19 as expected n = 32
Motivation +0.34 better than expected n = 32
Exercises +0.06 as expected n = 32
Readings +0.05 as expected n = 19
Other activities +0.16 as expected n = 25
Pace +0.41 too fast n = 32
Difficulty +0.34 too hard n = 32
Novelty +1.06 mostly new n = 32
Read what they wrote39
Worked well12
  • The claude code exercises
  • coding exercises and workspace management
  • The exercises helped, must most of the exercise (for me) was spent trying to understand the fundamentals, so I didnt really progress to solving the actual problem. The backdoor exercise was informative.
  • The morning exercise discussion was excellent.
  • The mechanical and coding exercises today made it very clear what was going on and helped solidify the concepts because I was able to take time and let the concepts sink in or ask Claude to link concepts I didn't get on the first pass.
  • Stegonography protocol exercise
  • The balance across theoretical treatment, empirical results discussed at a high level, and hands-down herdr exercises worked really well for me.
  • A good mixture of the materials both in theory and practice. The interactive discussions between the participants were especially helpful to understand the concepts thoughout the lecture. A good engagement with the lecturer was also a plus.
  • Reading and exercises
  • I liked recreating the paper in the end.
  • I enjoyed the exercises, we spent time thinking about it in groups. Perhaps we could do more exercises, but timed for sure to keep everyone on track.
  • The presentation was mostly clear and understandable.
Would improve16
  • Present the definitions of concepts before using them
  • more claude code exercises like the one we had in the end,
  • Material was very interesting, but I had no experience in cryptography at all, and much of the set up and intuition seemed to rely on this. I would have appreciated a brief intro to some of these methods in isolation before applying them to the LLM. For example: I didn't initially realize that taking M XOR K was a form of encryption until someone pointed this out to me. Taking K to be a random variable was another non-trivial extension. Applying this then to the LLM was confusing, because by then I was still unclear on the fundamentals. I recognize however that this is because of my unfamiliarity with the background material. I'm sure that someone with a stronger CS background would have followed just fine.
  • Learning Claude vibecoding should be well-separated from learning the lecture material, or I have to do through a "boring" exercise to get to a useful one
  • I could have used much more guidance with the downloads. I thought I had set it up before the Intensive, but was so slow in matching the various software I feel way behind. So maybe a pointer to what, exactly, should be set up before the afternoon session.
  • Would appreciate much slower pace for foundational material when introducing names of variables and then speedier overview of derived concepts.
  • The coding exercise in the first half could have been ordered to be before the derivation or maybe both could have been given at the same time. Also a lot of people around me spent a lot of time installing Claude Code and other tools so a heads up about it would have sped up the exercises considerably.
  • Would be nice if we could learn more concrete examples on steganography and backdoors, what are some common protocols out there (and how they work in practice). How do "harmful responses" produced by LLMs relate to hallucinations etc.
  • First half the content was cool, but it was a bit hard to understand! Motivation could have been clearer. But good overall
  • Maybe some more structure prerequistes would have made the session better. It would have been good to have the coding tools like herdr working beforehand
  • The slides were a bit chaotic imho, and we jumped into the maths fast, while an intuitive concrete example would have been useful at the start :)
  • The coding exercises could be truncated.
  • I think the presentator was really fast
  • In a lot of the slides about Chain of Thought monitoring, a bunch of graphs were shown with no explaining text. It made it therefore difficult to follow if I missed a sentence the instructor said. Otherwise, it was great!
  • The explanation of the optimal steganography algorithm with a shared source of randomness was unfortunately very unclear. I spent a lot of time understanding it after it was given, and I have the impression that most attendees simply gave up. It would help if the explanation was rehearsed, and if the instructor had a dedicated place to write (or had slides with pauses prepared) to give a step-by-step detailed derivation. In the end the concept is simple to explain, e.g. with pseudocode, so it is really a loss that is was presented in such a confusing manner.
  • I would improve the Claude code component. I think it was a little too open-ended. Perhaps asking us to replicate a specific figure would be better. Also I think the presentation on steganography would be improved by including a very concrete example that instantiates all the variables and such.
Changed my mind9
  • I was already familiar with the main concepts as information theory and I enjoy to learn the many new things we went through.
  • Steganography can be used to secure a system, not just a threat to AI safety
  • I was skeptical about text watermarking working for frontier models without it degrading performance, but after today its seems possible, mathematically speaking at least.
  • I did not reallised essentially perfect steganography was possible with LLMs
  • I learned a good amount of knowledge about how watermark works in LLMs and the mathematical views of stegonography.
  • That i should care about encoded messages
  • I just did not know about steganography at all.
  • I really like this topic, too! I would like to know what open problems in this area are important.
  • No
Other thoughts2
  • Julian delivers content well. Thank you!
  • None
Mysteries of Deep Learning Tue 13 Oct, Charles Renshaw-Whitman 37 responses · energy 3.8/5
Overall -0.66 below expected n = 35
Content -0.17 as expected n = 35
Clarity -0.40 below expected n = 35
Motivation -0.20 as expected n = 35
Exercises -0.05 as expected n = 22
Readings +0.06 as expected n = 34
Other activities -0.15 as expected n = 13
Pace -0.14 about right n = 35
Difficulty +0.17 about right n = 35
Novelty +0.51 mostly new n = 35
Read what they wrote44
Worked well11
  • the reading material seems really good and structured and i am very grateful for it
  • Serendipitously, I had wonderful discussions with other participants that helped develop my inituitions for how I fit into AI safety.
  • I greatly appreciated open discussion with Charles.
  • There was a lot of material, but it was well organized.
  • Having time to read and work in groups to digest the material was nice.
  • The readings and resources on the website were very well organized
  • The Github page is nicely made and provides a good backbone for individual learning (although, as mentioned, I would have preferred a little less individuality).
  • The assortment of the readings on the various mysteries of DL.
  • Nothing really honestly, it was just read stuff by your self I could have done it at home
  • I like being allowed time to read the material thoroughly myself
  • For the whole day I read one of the suggested papers and reproduced one of the results with the help of conding agents. That worked out.
Would improve25
  • while the resources made available are extremely valuable, i really do not think that this format of "go and read whatever you want from this way too big corpus" is a good use of our time. i would have preferred some of the material be presented. Mysteries of Deep Learning, Training Dynamics, and Singular Learning theory are the days i was most looking forward too and I was very disappointed to hear that today was going to be this format and VERY disappointed thinking the other days are going to be like this
  • i think there needed to be more direction for mostly unfamiliar material
  • In the beginning of class, there was a brief discussion of what each section was about: I would expand on this a bit and give everyone a bit more of a sense of the major papers in each section.
  • I appreciate the collection of resources provided. Lecture format would have been preferred during the day.
  • The morning session was a bit unorganized. We basically had to explore several topics ourselves. Would be nice to have organized lectures around the most important topics so that we could at least get the high level ideas.
  • More structure given the fact that the topic I very broad. I don’t mind reading but I prefer it if it is aimed at something specific.
  • Though I appreciate the thought in a self-guided philosophy, I would strongly prefer a greater structure and greater input from the instructors. I can read curated papers and ask LLMs for advice on further curation in my own time. The only benefit of attending today was the presence of other participants. Note that I had trouble answering questions about pace, clarity, and difficulty given the absence of pacing, explanation, or challenges.
  • The format of the day could be improved, in my opinion. Specifically, I would have appreciated a more selective list of articles, with some structured discussion groups among the other participants (and Charles and Brianna). Ultimately, I find it much more useful to engage in discussion than quietly read articles, to the extent that the discussion doesn't dwell in too much bullshit.
  • It was all self reading today, so a heads up on Discord would have helped prepare us better for that.
  • I am a self-directed learning aficionado and I appreciate the github repo, but this is not what I traveled to SF, and commuted to Mox, for. I paid someone 1k today to do my job at 5 just to have the flexibility to stay here, and I am not sure for what. At least you could let us know in advance that's the format. The insane amount of material was also discouraging. It's the responsibility of a teacher help students make the most of the time when the amount of material is too high. It is one of the very unique few things an instructor can provide.
  • A bit more structure to the day. It would have been nice to have 15-min check-ins with a brief intro or overview of the modules during the day. The readings were also a lot; the check-ins would have helped highlight what was important to get and their relevance to later sections.
  • I would appreciate a high level overview of each of the 3-4 mysteries before diving into the readings. I think it remains important to distill some intuition/opinions to overall get a sense of direction or engagement with the research questions before diving into it ourselves.
  • I am not a big fan of the 'just read some stuff' approach. I personally need a more concrete approach and a stimulus to engage in the material with fellow students. The current approach made me drown in the freedom somewhat, which is a shame, because the material is genuinely interesting.
  • Might try a little more structure to the day. Perhaps two 20 minute talks to anchor the rest of the day's readings.
  • I understand the motivation behind giving us a repo for self-study, but what I value most in a lecture is a clear throughline connecting the most important ideas, even if that means being selective and leaving some material out. I was looking forward to these lectures as a chance to hear how researchers in the field think about the topic and which ideas they consider most useful. Discussion and questions are valuable too, but I think a prepared lecture gives the speaker more room to develop their perspective in depth than responding to questions on the spot.
  • Unfortunately, the lecture did not really take place. I believe many students, myself included, were genuinely looking forward to it
  • I think a general structure was missing and a better outcome could be achieved with more structure.
  • Make it a class
  • not much maybe a few lectures but i like self reading
  • Maybe a Bit more teaching in the classical Sense, content was good
  • Give an overview of the ideas before delegating the reading
  • Unstructured reading time does not feel productive, that is something we could have done remotely (outside of the intensive).
  • Required more structure: free-form from the get-go left a lot of people confused. Perhaps spend the time before lunch going over some basics and giving us some intuitions before setting us free.
  • It would be great if we get a general overview of the topic during the initial hours: The materials were good and the session gave an opportunity to dive deep into the topic of interest. Charles and Brianna were around all day to answer all questions. But a s a person who was new to this field, reading 'alone' was not really helpful to actually uderstand the topic well. I would recommend strongly to include bit more lecturing hours - say 1 hour in the morning and maybe 30 mins in the afternoon. This would help us steer into the reading sessions.
  • The formation of the lecture. I would have liked much better if the lecturer provides substantial materials explained.
Changed my mind6
  • I think I am convinced that research into aspects of deep learning or interpretability for transformers are not for me. These directions do not address fundamental concerns I have (total loss of control, gradual disempowerment, bad actors), while being less aesthetically pleasing than other research opportunities.
  • I'm doubtful Singular Learning Theory is a promising approach to native LLM ontology.
  • I was better able to understand why even measuring generalization is a difficult problem and how counterintuitive deep learning can be on that front.
  • I became a bit more aware of the magnitude of stuff for me to read/learn and it increased my drive to dive into things.
  • Virtually everything was new, so I did not change my mind.
  • No
Other thoughts2
  • I think the self directed learning approach was ineffective. There were too many readings to chose from and I didn’t know what to prioritize.
  • Please try to implement the suggestions about lecturing hours pleeeeease!!!!!
Physics of Deep Learning Wed 14 Oct, Tudor Dimofte 26 responses · energy 4.0/5
Overall +1.13 better than expected n = 24
Content +0.88 better than expected n = 24
Clarity +0.63 better than expected n = 24
Motivation +0.42 better than expected n = 24
Exercises +0.54 better than expected n = 24
Readings +0.67 better than expected n = 24
Other activities +0.47 better than expected n = 19
Pace +0.25 too fast n = 24
Difficulty +0.33 too hard n = 24
Novelty +0.58 mostly new n = 24
Read what they wrote28
Worked well10
  • Pacing ws excellent
  • I enjoyed the lecture format.
  • The lecture was super super clear, well explained, and nicely paced.
  • the structure was really good
  • This content was exactly what I came to the intensive to learn. I wish we had more lectures on exactly these topics. Readings and exercises were well selected.
  • HELL YES!! clear, structured, well paced lecture on fascinating and difficult material. this is all i wanted from this day and i wished more of the days were like this. perfect mix of lectures and exercises and discussion too
  • The schedule and lecture materials were well planned.
  • The excercises.
  • quality and clarity of presentation
  • The format of the delivery was excellent: excellent choices were made about what to cover. Also, the exercises generated productive discussion and advanced my understanding of what is going on.
Would improve12
  • Was excellent
  • This was interesting in the sandbox/toy setups used to study mathematical properties of NNs/LLMs but I didn't end up with much optimism about its application/potential to better understanding actual applications to LLM research.
  • nothing.
  • There are sessions that feel too open-ended, leaving too many things to explore. Would be nice to switch back to structured lectures.
  • This was the best lecture of the intensive so far. I wish we had this Tudor teaching for another few days. There is literally nothing I could suggest to improve it.
  • nothing
  • It felt like the first two lectures were very dense but the third lecture was actually pretty light. I think the time could be distributed so that there was a little more time spent on the same material in the first two lectures and less on the third.
  • It would be helpful if lectures included explanations of relevance to AI safety & interpretability, both short-term and in the long-term.
  • The lecture, exercises, and notes were very well organized. Can't think of anything else to add within the time limit for the topic.
  • It was wonderful lecture!
  • nothing comes to mind
  • More time to digest the many ideas present here.
Changed my mind6
  • I had no prior knowledge of physics inspired neural nets of deep learning. This lecture inspired me to learn deeper into it.
  • I enjoyed the content today.
  • I have more hope that a theory of learning mechanics is possible.
  • Haven't changed my mind on anything really.
  • I'm slightly more convinced about learning mechanics being achievable at some point in the future.
  • unsure of how i can fit in when ive only done my undergrad
Singular Learning Theory Thu 15 Oct, Brianna Grado-White 35 responses · energy 3.8/5
Overall +0.34 better than expected n = 32
Content +0.28 better than expected n = 32
Clarity -0.38 below expected n = 32
Motivation -0.25 below expected n = 32
Exercises -0.03 as expected n = 30
Readings +0.16 as expected n = 19
Other activities -0.05 as expected n = 19
Pace +0.44 too fast n = 32
Difficulty +0.69 too hard n = 32
Novelty +0.75 mostly new n = 32
Read what they wrote40
Worked well11
  • I think walking through concrete examples applying SLT was helpful in getting the concepts to connect
  • This working lectures were clear, and quote advanced
  • I really enjoyed Lin’s talk, I think it was the most mathematical-audience oriented talk so far and catered well to my knowledge and familiarity. Lin gave some of the best explanations of grokking, double descent, and the staircase I have seen so far. He also had very good suggestions for questions to pursue (in fact I wrote up one of them for my project proposal!)
  • The slides were good, once I really understood a bit of algebraic geometry etc.
  • Shaowei's talk.
  • i really appreciated the first speaker's presentation. it was more mathematically involved than i could follow at times, but i really appreciated the clarity and structure, and how we went from simple statistical models to algebraic geometry and back
  • the exercises were excellent. the content was awesome. the discussions with colleagues and the lecturers were great.
  • lectures were thorough, clear, and interesting.
  • The lecture slides were great, perfect balance between math and visual/organized info.
  • lecture quality and clarity
  • The exercises were good
Would improve21
  • I think today would have been a bit easier to follow had the broad motivations been presented earlier and upfront. Instead I think it followed a route of introducing things as they were developed which was difficult given so many levels of abstraction and new concepts.
  • I thought both lecturers were great, and the subject matter was very interesting. This was one of my favourite days so far. I was very impressed and I think I learned a lot from them.
  • I really really enjoyed today! The one comment I would leave which is very specific to today is perhaps having a brief interlude where the connection between the morning talk by Lin and afternoon talk by Dmitry is explained! They were both using the word singular models but meant actually very different things by it. In fact in the morning Lin said most models are singular but by afternoon Dmitry said most models are regular!
  • As a non-math person, I found the contents a bit heavy and difficult to parse and used Chatgpt extensively to understand. I would really exactly one thing of the lectures - If we could have the slides beforehand, that would be idea. The people not familiar with the background can look up and try to make sense of the pre-requisites.
  • The Slides for the second Part we're Not ideal, maybe a more rigorous writeup of everything (both First and second Part) i.e. that goes into more depth and is less handwavey (or pointers to papers that are rigorous)
  • Catalogize materials in terms of prerequisites maybe? I don't think my path to impact lies in becomes a physicist and then applying phisics to ML. But then I struggle to motivate myself to work through this material and I don't know when it's worth paying more attention. (I've already wasted years studying math I never used and forgot.)
  • The post lunch lectures could be a bit clearer.
  • Better slides in the second half.
  • I think today's lecture might have been improved by breaking them all in half and having mini-exercise sessions in between all of them. There was a lot of theory here and it was easy to get lost; having breaks to catch up and do exercises more regularly might have helped.
  • The lectures are good. Would be nicer if we can learn a bit more about the recent literature on singularity theory and the open questions that people are interested in.
  • Prepare the slides better (Dmitry). First speaker was excellent.
  • i think having two speakers, one presenting big picture and one working things out in more detail is a great idea, but the second speaker's lecture was really hard to follow in the sense it was not clear to me what was assumed and what was shown and in general where we were going overall, i think it would have been helpful to spend some time motivating why we should care about SLT
  • Maybe the second half can be better structured
  • both lecturers had great insights to share. however, Dimitri could have improved the preparation, and polished his slides more, for a more efficient impact on me.
  • The first part of the day was engaging. But the second half didn't really connect with me. Also, both sessions were lecture heavy. Would be better if there is a balance with exercises and coding.
  • nothing.
  • The last lecture was hard to follow from the slides.
  • Not much - I really enjoyed this one.
  • both talks were good!
  • Maybe some more preparation of the presentation for the second half.
  • The second part of the day, while it was very very interesting, was a little bit hard to follow. Maybe, for the second part of the day, a possible improvement would be to concentrate on one specific result and focus a bit more on the implications for safety.
Changed my mind7
  • Loss isn't always the best measure of how a network is developing and generalization is an important thing to measure for capability development.
  • I think today convinced me that my particular skillset (the kind of math I’m familiar with and do) can be used to understand key questions in ml theory (grokking for instance!). I’ve walked away from this talk seeing good directions in AI Safety I can apply my knowledge to!
  • Mostly new, so no.
  • SLT seems less impactful, after today.
  • got more motivation into research of the mixture in both theory and practice for AI safety and MechInterp.
  • I was told that SLT was substantially algebro-geometric, but such seems like an overstatement. Nevertheless, there seem to be some cool problems to solve related to the content from today.
  • I've since become very motivated to study SLT, and am considering researching it in the context of math grad school or upskilling in the research engineering side, or both.
Other thoughts1
  • I was a bit low energy, so I might not have the usual level of energy to follow and understand.
Training Dynamics Fri 16 Oct, Charles Renshaw-Whitman 35 responses · energy 3.9/5
Overall +0.91 better than expected n = 34
Content +0.91 better than expected n = 34
Clarity +0.88 better than expected n = 34
Motivation +0.82 better than expected n = 34
Exercises +0.29 better than expected n = 28
Readings +0.25 better than expected n = 24
Other activities +0.61 better than expected n = 18
Pace +0.09 about right n = 34
Difficulty +0.18 about right n = 34
Novelty +0.47 mostly new n = 34
Read what they wrote40
Worked well13
  • the lecture by Daniel Kunin was totally amazing: clear and inspiring and presenting a compelling argument for why training dynamics matters for safety. the part on the whiteboard was especially great!! and i wish there had been more of this during the intensive. it forces the speaker to progress at “the speed of thought” and we can all follow along
  • The second lecturer
  • The morning exercises were a great preparation for the afternoon's lecture.
  • The level of mathematical formalism was was tuned just right (for my background/taste) so it it did not obfuscate the larger point.
  • The first morning session could have been more lecture structured. Otherwise, the second half was great.
  • Guest lecture by Daniel was fantastic. Very clear and interesting.
  • Daniels lecture was one of the best talks ive heard
  • Discussion and drawing on whiteboards.
  • Both Brianna and Daniel were excellent in their delivery of the information. The exercises, as well, we excellent.
  • I thought working at the board was good.
  • The lectures were well prepared.
  • the white board discussion and motivation via toy models was perfect, I would encore more speakers to attempt this format, it lands well with mathematicians and physicists
  • Nice to start off with the basics. Doesn’t hurt to repeat some stuff from time to time.
Would improve17
  • It was great today!
  • Teaching was excellent (in the beginning it was very high level, but a good overview)
  • I found a lot of this stuff hard to motivate from a safety perspective. I think some more motivations would help me stay focused.
  • it would have been helpful to know that the morning lecture was presenting things that were going to be used in the afternoon lecture. Instead, I got the impression that the content of the morning lecture was irrelevant in a very broad sense. I remain a puzzled by Charles’ and Brianna’s approach to teaching in the intensive. the votes below are strongly influenced by Daniel’s amazing afternoon lecture
  • Very engaging. The split between lecture time and exercise were ideal.
  • Maybe more talks today would be helpful! There was a chunk of time where I wasn’t quite sure what I should dedicate to.
  • First lecture with more info and more precise
  • nothing much, the white board presentation was extremely nice
  • For today, a bit more of the big picture and big results upfront to motivate what is going on later and guide exercises. I think also the afternoon's derivations could have been sped up had it been made explicitly clear during the morning that exercises were crucial.
  • Not much
  • Almost nothing to improved. Perhaps some embedding GIFs displaying function space changing through training epochs
  • the content was kinda light in the morning, and kinda compressed in the afternoon. that could have been distributed more evenly. also, some guidance or coordination for doing the exercises could be helpful, so there are groups to collaborate with.
  • I think one of the lectures went quite long and a break could have been good, but it seems like people were excited and interested.
  • Both lectures today were amazing.
  • The second half was a bit dense and didn't have any breaks. There was about an hour left afterwards so the time could have been better utilized.
  • Not much, Daniel was a great lecturer.
  • Tiers for difficulty (easy/hard)
Changed my mind10
  • My main takeaway from Daniel’s talk was that initialization matters, I’m still trying to parse through exactly how initialization affects the output in transformer based networks as opposed to the linear networks he gave the talk on.
  • I feel more that there's a bag of 5 observations that a lot of people are coming up with increasingly divergent theories to explain. I genuinely enjoyed learning the material today but it makes me think there are higher priority areas for research when it comes to AI safety
  • Mysteries of Deep Learning and Training Dynamics have emerged as the central scientific interests for me.
  • Learning mechanics seems like a very promising theory for Alignment from first principles.
  • Lazy vs rich learning. Studying these with higher dimensional parameter spaces.
  • I finally understand what's up with Lazy / Rich regimes.
  • I wasn't too interested in linear networks before today but I think I will look into them a bit more.
  • Daniel's lecture on rich learning completely changed my mind about how generalization happens during optimization. This was a very fun lecture to watch.
  • Daniel mentioned that the black box nature of LLMs comes from the data and not the algorithms, and that makes a lot of sense in hindsights.
  • It was mostly new.
Computational Mechanics Mon 19 Oct, Paul Riechers 34 responses · energy 4.0/5
Overall +0.65 better than expected n = 31
Content +0.81 better than expected n = 31
Clarity +0.42 better than expected n = 31
Motivation +0.45 better than expected n = 31
Exercises +0.67 better than expected n = 30
Readings +0.33 better than expected n = 18
Other activities +0.41 better than expected n = 22
Pace +0.06 about right n = 31
Difficulty +0.06 about right n = 31
Novelty +0.94 mostly new n = 31
Read what they wrote38
Worked well12
  • I really liked the exercises. They helped provide intuition for both the morning's lecture and foundation for applications in the afternoon.
  • I think the exercise of constructing the belief states was useful.
  • this was an amazing lecture! i particularly enjoyed how Paul spent all morning on the basic notions needed to understand "the meat" ie his actual research work. i think this would be something that increases the effectiveness of most days: spend the morning on very basics anyone can follow, and then progressively move into more superficial presentations of more advanced results
  • the speaker was great at answering questions
  • Since the papers by simplex AI are popular in AI Safety circles, I’ve come across them before but to have Paul explain give a whole day of talks and exercises to understand the core ideas has been so enjoyable! I have a much better grasp of the idea now than I did skimming the papers earlier. As always I enjoyed the exercises and learned the concepts better through them.
  • Content was very interesting
  • exercises
  • The exercise was certainly necessary.
  • amazing discussions within participants and a lecturer. Great environment to develop research ideas.
  • Paul, the lecturer was very nice and explained things well and interested in talking to us.
  • I really enjoy the topic, explanations and discussion with the mentors helped a lot.
  • Paul knew the concepts really well and the presentation was well designed.
Would improve18
  • Show slides 15s longer so we can read and digest the equations. Everything else was fantastic!
  • The lecture was really interesting, and I think this is a foundational topic that should be included in any introduction to the field. I feel like I now understand belief geometries well. I did find the tensor decomposition for the interacting agents a little confusing to follow.
  • one small thing that could improve the experience for the lecturers would be to give them a (possibly ai-generated) summary of the kind of background people have. a few of the speakers expressed uncertainty abt this
  • Paul Riechers talk was great, but would have benefited from more mathematical precision. A number of statements and constructions seemed to crucially depend on certain assumptions and these were not clear enough at times.
  • This isn’t quite something to improve but more of a suggestion, simplex AIs papers are somewhat known within AI Safety circles and it was so good to have them explained in detail today. Maybe big results/papers in AI Safety, as they come, could be a good lesson to dedicate an Iliad teaching day to!
  • Not much, exercises were very good as was the pacing
  • Have a little 5-minute break every 30 minutes to discuss a particular topic with our neighbour (for example very simply "explain what was just said"), which would help to digest the interesting (mostly new) information.
  • nothing
  • Not much, I really enjoy this subject and the slides have so much valuable information.
  • Would have appreciated more of a look at the mechanical aspects of the theory aspects of newer research papers in the area. Perhaps could have benefited from more structure.
  • I thought today was very good, but quite difficult. Perhaps this topic from Simplex's research could be spread out over 2 days instead of 1.
  • I only attended the morning session then it was lunch, then exercises until 2:15 which I didn't attend the evening session. But I don't think we covered experimental applications/findings of belief geometry to LLMs and the remainder of the slides until after 2:15 which constitutes like >60% of the material. I might've preferred a speed-run of the material upfront, and get to the upshots before lunch. Deriving the MSP and belief geometry for some processes is important but I feel like the main idea was already conveyed via white boarding in the morning.
  • Some more exercises would help. We only had one detailed exercise session today.
  • I would like to have slides accessible at the start of the day
  • Overall the lectures are nice. Would appreciate clearer definitions and examples of certain terminologies such as belief geometry and mixed state presentations.
  • Great lectures, nothing to change here. The material is a little sparse, it would help to have more information in the slides (links to the papers mentioned, etc)
  • Julian knows his stuff, but for first time consumers of the topics like me he assumes too much knowledge and underestimates slide-fatigue. I would encourage him, to observe how, e.g., Tudor lectured. The pace was slower, the audience was guided through each slide, all notation was explained, all graph axes were labelled, the context was repeatedly given, it was emphasized when things were important to retain and which were excursions etc.
  • Nothing, great lecturer today + good exercises.
Changed my mind7
  • I am more excited to dig into belief state geometry as a potential research project
  • This convinced me that belief states are the right way to think about transformers.
  • Computational mechanics seems like a really appropriate framework for looking at LLMs. The most convincing one I have seen so far
  • I did not know that HMMs were used in this field.
  • Using large language models to predict hidden Markov models produces accurate fractal probability simplicies. Transformers have an inductive bias to factoring into low dimensional components of the activations weights. Factors representing abstract features are generally found in early layers and the syntactical factors generally occur later.
  • I was not familiar with this direction of research and it seems really promising.
  • No. I have been interested in computational mechanics for a while so at least the conclusions were not new to me.
Other thoughts1
  • 2nd submission for this day by the way (I recalled some more feedback on my end)
World Models and Transducers Tue 20 Oct, Alec Boyd 34 responses · energy 3.7/5
Overall +0.33 better than expected n = 33
Content +0.27 better than expected n = 33
Clarity +0.55 better than expected n = 33
Motivation +0.21 as expected n = 33
Exercises +0.12 as expected n = 33
Readings 0.00 as expected n = 15
Other activities +0.25 better than expected n = 20
Pace -0.09 about right n = 33
Difficulty -0.06 about right n = 33
Novelty +0.45 mostly new n = 33
Read what they wrote39
Worked well14
  • frontal class with a lot of partecipation and questions from the audience plus nice exercises
  • Questions to the audience
  • The birds eye view in the end and applications to multi agent systems seemed like a good perspective on how such theories could be practically relevant for safety, even when we do not have a full theory
  • Possibly the only day so far where we were not rushing to go through content, and definitely the only day I was able to complete all the exercises in the alloted time.
  • the exercises went well for me, but I was not a fan of the material in general.
  • The exercises were very well-chosen to reinforce the theoretical foundations covered in the lectures.
  • Working together in subgroups on particular exercises. Enough breaks (and suggestion to go outside for a while). The presentation itself was just well-paced and I liked the simple examples to really get the hang of the processes ourselves.
  • great pedagogical approach, putting things into context, building the basics, and then eventually getting to complex applications
  • The lecturer was very good: kind, clear, engaging.
  • We discuss about a possible project together with the mentor. I am very happy about it and I look forward continuing working on this!
  • Some content reviewed Mondays lecture but I liked it, it gave time to properly digest the material, and also having 2 different people explain similar concepts gave nice perspective.
  • You might find this in many of my feedbacks, but I enjoy doing the exercises! During the time allocated to exercises, Alec was pacing the room and offering to help out (not that this is necessary, all mentors have made themselves very available) which made it easy to ask the simplest questions!
  • It was a good idea to introduce transcoders more formally. We had seen something like this the day before in Paul's lecture (where he constructed belief states using a Bayesian update rule), but this helped to make this concrete.
  • I liked the presentation style and motivation. The presenter was very clear and deliberate
Would improve16
  • nothing it was great
  • The lecture in general had a lot of overlap with the one from yesterday and hence at times was a bit boring/too slow, especially in the beginning However, the later exercises was a good reinforcement of the things we did yesterday
  • The concepts today felt more intuitive than that of Day 11 (Computational Mechanics), and would have helped to see it before Day 11.
  • This day had a remarkably large overlap with yesterday's topics. In some cases, it presented the details we wish we had the day before; in other cases, it felt like an unnecessary review. It is nice to spend two days on related topics, but they should be better coordinated together.
  • I was not a fan of the topic, so I quickly did not follow and got lost. Perhaps this topic is a little out of place and should be skipped?
  • It would have been nice to focus more on the implications when applying this analysis at scale for LLMs and factors/multi-agent systems
  • I only attended up to Section II - up to that point this was interesting but the upshot hit a bit less well as I feel like world model theory is already implicit in the computational mechanics lecture, so to a certain extent it *felt* redundant.
  • This felt too overlapped with the computational mechanics day. A lot of things (including the MSP exercises) were extremely similar to yesterday.
  • The lectures are nice but would like to see how the things we learned can be applied to understanding LeWM and other SOTA world models, not just toy examples
  • The lecture was well structured. Went from good overview to very new studies. One improvement would be to include more codes to practice. There was one exercise with coding. I would increase the proportion.
  • I would make the content of today more independent from that of yesterday.
  • this was great!
  • At times I felt we got a little too in the weeds and I was struggling to understand how this relates to the big picture of alignment and what we are trying to figure out to make AI go well. That being said, he was a great lecturer. The exercises were helpful, he asked probing questions, and he was very nice and easy to talk to.
  • Maybe some more coordination with the previous day to prevent overlap
  • This lecture was really useful, but it seemed like there was a lot of overlap with Paul's lecture from the day before. It was really helpful for me to see this again, and I think Alec made a really good choice by introducing the topic in the language of transcoders. I'm really interested to understand what this formalism has to say about multi-agent interactions, and I'm looking forward to hearing more later on.
  • I think this lesson was redundant from yesterday
Changed my mind8
  • I feel that studying multi-agent systems is a bit more feasible and am a bit more optimistic about practicality of breaking down the enormous complexity in AI systems into interpretable parts.
  • By this point I've already developed an interest in SLT/mech interp/computational mechanics, but am starting to get a bit overwhelmed by the material and want to focus on how I can realistically best make an impact in this field versus broadening my horizons even further then going back to my status quo. Given my more technical background vs other participants, I feel like my path to impact in AI safety would initially be in the research engineering route while keeping up to date with AI alignment research progress vs driving it theoretically. I *am* considering a MSc and research supervised by academics in SLT (already in discussion) but I feel like short-term I should aim to stay in SF and continue engaging with Mox/Iliad. As such, I've been applying to jobs (in data/evals/post-training) in SF and am currently in a couple interview processes, and hoping to informally re-attend future Iliad Intensive lectures and cover gaps in my knowledge in a longer time horizon if that's okay with the organizers.
  • Automata can actually be an interesting approach to understanding agents.
  • cool research! i was unaware of the material from yesterday and today and how interpretable this aspect is!
  • The multiple sub-agent picture seems like an interesting line of work for understanding AI.
  • Comp. mech seems very interesting, might be what I work on if I go for AI safety.
  • I was surprised to learn that the Geometry of belief states reflects geometry of prediction, and it had me thinking about Paul’s talk on belief state geometry the previous day. I haven’t had enough time to mull over this to create a project idea out of it yet, but I certainly have questions I’m thinking about!
  • I came away pretty confident that "transformers encode belief states in the final layer of their residual stream" is a solid starting point for understanding transformers.
Other thoughts1
  • just as a suggestion: there could be a day on autoformalisation -> ai-maths for ai-safety
Project proposal day Wed 21 Oct, Leonard Bereska 21 responses · energy 4.2/5
Overall +0.68 better than expected n = 19
Content +0.42 better than expected n = 19
Clarity +0.26 better than expected n = 19
Motivation +0.58 better than expected n = 19
Exercises +0.50 better than expected n = 8
Readings 0.00 as expected n = 4
Other activities +1.00 better than expected n = 12
Pace +0.21 about right n = 19
Difficulty +0.21 about right n = 19
Novelty +0.26 mostly new n = 19
Read what they wrote22
Worked well6
  • Today, it was the proposal day. It was very intense, I enjoyed it and it was very nice both the format of the activity and the synergy with other participants.
  • good sinergy with group mates
  • I found working on actual research proposal and running a small experiment very productive.
  • I think it was good having us write proposals and engage deeply with a particular topic.
  • It was great to have some time to dive deeper into a topic based on what we felt inspired by.
  • The sporadic collaboration on any research topic was surprisingly very fruitful! I had a lot of fun engaging with everyone and just seeing what problems people were interested in.
Would improve13
  • I think it’s amazing that we have a whole day to write a project proposal! I think the structure of the day works as well — first giving us time to write up the proposal, then elicit feedback/ peer review, and then finally submit it. This is not quite a critique of the format, but if we could have a little more time, perhaps until dinner or midnight to turn in the proposal as opposed to 5:30 it would be helpful! That way if we’re hoping to submit more than 1 proposal we have enough time to spend working with two different teams and turning it in! I think the time provided currently is definitely sufficient to write up one good proposal, I had to rush my second one but there was still enough time to get across the basic idea. It has been so fun discussing, writing, and reviewing ideas, definitely one of my favorite days!
  • I would have posted to guidelines the night before.
  • At better scoring system. The average of top two submissions means that if you have one good proposal and another you are not sure about then you don't necessarily want to submit the second one.
  • Maybe having a set of interesting papers that are recommended to work on would be good!
  • more time, id do something like 2 days.
  • The word counts of the proposal could have been clarified a bit more, especially as compression of ideas is often the hardest step. The point about how multiple projects would be scored was not clear as well.
  • In google docs the word count seems like a recommendation, while on the form it is a hard limit. Would have been good to know. Also to know the structure earlier in the intensive would have been better.
  • Announce the grading of project proposals well in advance and tell us to think about project during the intensive!
  • More feedback from experienced researchers. Clearer explanation of the intent. Was this an exercise or an exam?
  • I think 2 proposals was too much. It took me a long time just to get the 1 proposal right.
  • I felt that the peer review process could have been better organized by having someone come up and say it was time to review others' papers. Some were still brainstorming proposals or still writing when this happened which made it hard for those who had finished on time to incorporate review comments when the second wave actually got to them. Some also openly admitted to not reviewing anyone's proposals which felt a bit unaltruistic.
  • i have a bit of mixed feelings about this... on the one hand it's great to have time to think about a possible original contribution to the field, and I am glad to have gone through the exercise. on the other hand, it's a bit risky, from the academic-credit-attribution perspective, to assemble people into groups and have them pool ideas and pitch projects, without knowing whether people will have the opportunity to work together (say one person in a group gets into a fellowship with and the others don't, for example). maybe i am being too cautious or maybe in the end the benefits outweigh the risks.
  • Explain the rules of the gradient in advance
Changed my mind3
  • a little more time
  • I had some conversations about belief geometry that were very interesting and made me think that it might be even more promising a research direction than I originally thought.
  • After today the ambitious approach to AI safety grounded by theoretical advancements seemed a bit more feasible. Despite just having a relatively short intro to the material, classmates were able to come up with interesting questions that had far-reaching implications for AI safety, and I was a witness to that. On the flip side, I could also see where some research agendas while interesting had a very difficult time connecting with AI safety.
Data Attribution Thu 22 Oct, Charles Renshaw-Whitman 24 responses · energy 3.5/5
Overall -0.50 below expected n = 22
Content -0.27 below expected n = 22
Clarity -0.32 below expected n = 22
Motivation -0.14 as expected n = 22
Exercises -0.40 below expected n = 15
Readings +0.15 as expected n = 13
Other activities 0.00 as expected n = 5
Pace +0.09 about right n = 22
Difficulty -0.05 about right n = 22
Novelty +0.59 mostly new n = 22
Read what they wrote21
Worked well4
  • improvement on last lectures he has given as he gave somewhat of an overview before reading, bu could have given a more structured lecture
  • Nothing, really. Maybe the videos Charles spent 10 million tokens on are insightful, but he did not really present them in a way that raised our expectations much.
  • I think Charles is really interesting and always appreciate his takes.
  • I liked having lots of breakout time to walk through the derivations myself
Would improve13
  • Not much, maybe some dedicated slides?
  • Slides
  • The notes are well organized, but the lecture was not. Charles believe is self-directed learning too much for it to have worked.
  • I liked the initial lectures on the topic, but near the end some more guidance wouldve nice.
  • The lectures are a bit unstructured and unorganized. Improving the presentation and organization would definitely helps a lot for understanding the materials better.
  • Today felt like a wasted day. The other day that felt like this was the first day Charles was "teaching". I was optimistic to hear that he was going to alternate between lectures and exercises today, but unfortunately my optimism faded away within one hour, after Charles refused to provide informative answers to questions (rather than quips), failed to reproduce derivations on the board, and more generally had clearly not prepared a coherent presentation of the material. Most importantly, he kept undermining the material he was teaching as he was teaching it. The Intensive is described on the website as "A four-week, full-time taught course". I don't see why you would entrust so much of the course to someone who seems completely unwilling to provide good pedagogy. The course page for today also has pretty bad notation (β used both as the datapoint weights and as the inverse temperature).
  • I would add a bit more of structure. I like a lot the idea of learning by reading relevant literature, this was very cool! However, I would have appreciate a clearer program to follow, for instance to coordinate 30 min of reading and 15 min of discussion, or something on these lines.
  • Prepare a lecture, not reading a doc
  • I think data attribution can be really cool, and Charles is a fun and intriguing guy who clearly knows his stuff cold, but I did not learn much because of two reasons. First, I felt like we did not get enough time yesterday to really work through our project, and I found myself wanting to go back to refining it further, and more generally I felt like _doing_ something instead of another day of _learning_. Second, Charles seems like he does not enjoy teaching very much, and does not appreciate how important context is before moving straight to the math in a way that suggests that it's mostly trivial stuff that we would learn better on our own. And I'm sure that's the case for him, but that's just not the way that most people learn or get passionate about a new subject. So what would I improve: allow for more time for practical work (which many people chose to take spontaneously now), and get teachers to be passionate about teaching.
  • I think it is interesting in and of itself, but it seemed like data attribution is not generally used by frontier labs to improve models, which I thought was unfortunate. It wasn't entirely clear what place it has in larger machine learning research.
  • Thank you for taking into account the previous suggestion to include lectures. They helped me a lot to get the overview of topic. The topic itself was bit far from my expertise. Despite that I got an impression of why this is done. I would include few coding exercise too.
  • I think Charles did a good job showing us where the topic fits into the bigger picture
  • I would have spent a little more time on notation in the beginning. The lecture was pretty rapid on math and it made it hard to follow without grasping the notation.
Changed my mind3
  • I did not really know about this topic beforehand.
  • I think model susceptibility under data perturbation is a fascinating idea. Wish I had more time to look into it.
  • This really got me to think more rigorously about how to do attribution at different levels/scales within models and systems. However, I think it was a bit general and would have appreciated seeing how it gets used in practice to make the connection to AI safety clearer.
Other thoughts1
  • Let me be the person who raises some uncomfortable criticism. I believe the lecturer is knowledgeable and competent, but I would like to share a few concerns for the sake of improving future programs, even at the risk of becoming a less favorable fellowship candidate. 1. In a professional academic setting, I found the frequent use of F-words and certain jokes inappropriate. They made me uncomfortable and affected my trust in the learning environment. I spoke with a few others who shared similar concerns to varying degrees. 2. While no doubt that the lecturer clearly understood the physics, I struggled to see how the theoretical material connected to AI applications. This may partly reflect my own limited background in physics, but more concrete examples, motivation, and explicit connections to AI research would help participants from different disciplines stay engaged. 3. Some responses to participants' questions felt dismissive. Rather than carefully understanding and addressing the questions, the lecturer sometimes responded with jokes or answers that did not sufficiently address the point. I believe respectful engagement with questions is especially important in an interdisciplinary program. To be fair, I attended only the morning session and spent the afternoon working on our own project. My observations are therefore limited, and later sessions may have been better. I take this program seriously and want to make the most of every minute here. I have no personal disrespect for the lecturer, and my criticism is not about his expertise but about the learning experience. I am sharing this because I genuinely value the program and hope the feedback can help improve it.
Multi-agent Dynamics Fri 23 Oct, Aden Power 27 responses · energy 3.5/5
Overall +0.26 better than expected
Content +0.30 better than expected
Clarity 0.00 as expected
Motivation +0.37 better than expected
Exercises -0.09 as expected n = 23
Readings +0.20 as expected n = 10
Other activities +0.13 as expected n = 23
Pace +0.26 too fast
Difficulty +0.15 about right
Novelty +0.67 mostly new
Read what they wrote28
Worked well8
  • The mix of various speakers and the intuitive approach
  • I liked the subject matter of James' lecture the most. All 5 talks were impressive.
  • It was helpful that different lecturers have their own expertise. Topics are various but coherent. Interesting topic and discussions.
  • I enjoyed the multi-agent RL talk!
  • i loved that the day was so well curated with a coherent programme±
  • Final discussion
  • The exercises with accompanying reading material really helped solidify my understanding today
  • It was best when they slowed down for questions and encouraged audience engagement.
Would improve14
  • The remote speaker was a bit hard to understand, both physically and content-wise.
  • Bad slides, not helpful or informative to read them during the lecture. Better preparation overall would be nicer, the speaker just read off the slides all the time and they were not written well.
  • Not teaching related, but the room was so hot today. Some fans would help a lot
  • I thought this was a really interesting topic, and particularly enjoyed James Sykes' talk. I would have really liked to see more through the lens of condensed matter physics. I'm now significantly more interested in this topic than where I started.
  • More exercises
  • Very good set of lectures. The lectures were organized well and covered a lot of topics. The lectures were clear and the discussion session a tthe end was especially good..
  • Today's lectures are very nice, and the exercises are good too. Nothing that I can think of for improvement, just wish today is not the last day of the week where we get a bit tired.
  • The online talk was a little confusing and very detailed
  • Make sure beforehand mic and the screen work.
  • I was a bit under the weather today, hence missed some things, my main gripe would be it wouldve been nice to have more of a concrete connection between the talks/cross references. But it was in general very nice.
  • two of the talks (James and the second one by Aden) went a bit too deep on the formalism and maths and specific examples and not enough about basic formalism (the Aden's first talk was great! but did not prepare us for James) and general ideas.
  • Not much, maybe slightly less lecture time. Also fix the AC
  • The lecture needs to spend more time getting us familiar with the notation before sprinting through successive math equations. It is hard to context-switch day after day between different conventions across different fields and coherently follow things. I think comprehension would be greatly improved with just a tiny slowdown in the beginning of the lecture.
  • I thought it would have been perhaps been better to start with the real world examples and discussion around them earlier in the day. Having it before the theory stuff would have put the theory in more context I think and perhaps easier to follow.
Changed my mind6
  • I learned about deal-making - fascinating!
  • This seems like a more interesting topic than I realized.
  • This subfield seems particularly new that there are not well established theories based on math or physics at this moment mostly empirical and engineering which could be seen as open opportunities or exploratory. An important field which could benefit the most from contributions in cognitive science or computational psychology aspects imo. Exploring with RL concept was particularly intriguing.
  • Enjoyed thinking about game theory, might investigate further
  • I changed my views on studying cooperation and altruism as emergent and defined behaviors in systems. I thought it had to be a product of behavior but I am now more open to mechanistic or training-based answers.
  • Interested in what the orthogonality thesis means. Realizing I may have misunderstood it.
Debate and Provably Safe AI Mon 26 Oct, Aden Power 33 responses · energy 4.0/5
Overall +0.42 better than expected n = 31
Content +0.42 better than expected n = 31
Clarity -0.10 as expected n = 31
Motivation +0.32 better than expected n = 31
Exercises -0.03 as expected n = 31
Readings +0.11 as expected n = 28
Other activities 0.00 as expected n = 30
Pace +0.10 about right n = 31
Difficulty +0.26 too hard n = 31
Novelty +1.00 mostly new n = 31
Read what they wrote41
Worked well13
  • Aden is very engaging and knowledgable.
  • I liked the discussion on complexity theory, but I would have liked to gain more intuition from it.
  • Small post-reading debate among fellow students
  • liked the presentation and the exercises.
  • The handout was nice to reinforce the more difficult proofs
  • exercises and discussion
  • The exercises and demos were very illustrative
  • Today felt very interactive!
  • I thought Aden's style of presentation was very good, the exercises felt relevant. The pacing was perfect imo.
  • The reading
  • The day was well structured and planned.
  • Aden's enthusiasm and willingness to teach! I really liked the exercise of reading the papers and recording at each step how our beliefs changed. It was a simple, almost gimmicky thing but it helped frame the reading!
  • The theory was very well explained
Would improve18
  • I thought the lectures were good and the topic was interesting--I think I had trouble following just because my energy was low. But I am super grateful for the material and will look back on it at a later date.
  • The mathematical formalism was interesting and I'm glad I saw it, but I'm not sure if it helped me to better understand how agent debate works. This picture gave me a different understanding of what debate is (in terms of complexity theory) but this didn't change my empirical understanding of it. I'm new to the topic though, so I might have missed something.
  • It was quite excellent! Perhaps speak a bit more slowly with less filler words like "uhm" and "like", but overall I like Aden a lot.
  • The lectures are overall quite nice. Probably explaining at a slower pace would help for those of us who are not familiar with the topics.
  • A little more details on how to actually implement debate in training
  • The lecture was good. The content gave a good overview of the topic.
  • I'd trim away the proofs, use more figures, and begin the lecture with the Alignment motivation.
  • The connection to how this would be implemented in practice or what are the practical implications could have been clearer. But still very interesting topics.
  • The lecturer made some mistakes that confused people
  • A bit less time introducing complexity theory, a bit more time on later topics, especially the prover-estimator debate towards the end.
  • Fix a couple mistakes in the slides that created confusion
  • Maybe a bit more emphasis on the connection between theory and empirics for debate would have been helpful today.
  • Only a small wobble on notation
  • Maybe some more preparation for the presentation
  • I wish the topics were better split between the theory and empirics of debate as a means for scalable oversight. The theory is just so far from how it's actually implemented that it is jarring to go back and forth.
  • Some of notation was confusing as presented, but it was eventually clarified in the QnA.
  • Despite his enthusiasm and passion, Aden seemed a bit out of his depth on this day. I would suggest improving the slides and the presentation in general by doing a more standard definition-example-theorem-proof kind of presentation (many slides didn't have definitions, theorems didn't list assumptions clearly etc) and then of course prepare more what is going to be said. I would suggest presenting and thinking of examples of the different classes and present them immediately. There _must_ be traditional and effective ways of presenting this material! Also I would totally introduce and rely on the Turing machine as the backbone of all this
  • The theory was nice, but the connections to how debate is implemented in practice were not fully clear to me, i.e. how this formalism translates to the way it is done in practice/how these are useful.
Changed my mind10
  • I am more amenable to thinking debate is a viable option.
  • I did not realise there were so many complexity classes
  • Didn't know about the debate approach to safety; seems immediately-applicable.
  • complexity theory has interesting points on AI safety, which is cool
  • I am really excited about ai safety via debate+scalable oversight and would love to pursue it. I have a strong background in computability theory, decision theory, and logic so this approach to ai safety seems very interesting to me
  • I am a bit more open to the idea that debate and estimation have a place in AI Safety rather than being pipe dreams.
  • I think debate is a more valuable research area than previous.
  • Debate is more important than I thought
  • I am a little less skeptical about debate working as an AI Safety technique, though not yet convinced.
  • i didn't know debate had such hardcore CS-backbone! Of the hard maths we saw so far, it seems the one with the clearest application to scaling and safety in practice (even though the connection between theory and practice is still kinda loose)
Agent Foundations Tue 27 Oct, Aden Power 26 responses · energy 3.5/5
Overall +0.25 better than expected n = 24
Content +0.25 better than expected n = 24
Clarity +0.17 as expected n = 24
Motivation +0.08 as expected n = 24
Exercises -0.22 as expected n = 23
Readings -0.36 below expected n = 22
Other activities +0.10 as expected n = 21
Pace +0.04 about right n = 24
Difficulty -0.08 about right n = 24
Novelty +0.75 mostly new n = 24
Read what they wrote22
Worked well6
  • The conversations about Agent foundations and where it stood in relation to the rest of the field were very enlightening. Aden does a really good job explaining.
  • The discussion part
  • Aden's best day! loved all of the content he presented
  • I really like Aden's pedagogical style.
  • The toy models/games given during the lectures really helped illustrate concepts around agents
  • Aden's enthusiasm for the topics is contagious. Even those topics that I came in somewhat uninspired about, he helped me stay motivated.
Would improve11
  • Very good presentation, can be more structured.
  • I wouldve liked this day to be a bit more focused on the different research agendas, it felt a bit rushed as a whole. All the topics where interesting, i just wouldve liked more time on the foundations.
  • A lot of terminology was introduced for intuitive concepts which eventually became hard to remember and follow. Not sure if there is a way to reduce that, but that would have helped.
  • I really enjoyed the conversations we had about modeling things as agents versus actually beleiving they are agents. It would have been nice if we could have had that conversation a bit longer.
  • There is not enough time for the exercises. For me, personally, it would for example be more useful to have discussion time, explaining topics we just learned to one another.
  • this was super great! the guest lecture was so so (too slow to build the definitions, and not enough time to talk through the actual result and relevance of it) (that's not why all my scores are not max)
  • I would like to see a clearer connection between existing theory and practice if possible.
  • I did not like the guest lecture. Getting bogged down in details before explaining what it matters. Foundations need solutions on lower philosophical level, not more formulas. Also, first lecture didn't seem to know what it's talking about in the end. Lob theorem is not a "subfield" of foundations, and tiling is not inextricably tied to this theorem.
  • I wish this day were earlier in the intensive, particularly before multi-agent day. I think it also would have been good to clarify more why we care so much about copies and how we think about copies.
  • The lectures are good, just like yesterday. A lot of materials feel too theoretical in nature, and it would be nice to have hands on coding session to validate the concepts we learn and to see if they are really useful in practice
  • i think the guest lecture didnt really match the flow of the rest of the day and threw everything off
Changed my mind4
  • The embedded agents framework seems like a promising research idea. I didn't realize it was so underdeveloped.
  • I especially liked the final part, and I learned that some deep philosophical thought experiments have now become real-life problems, e.g. issues with perfect copies of agents.
  • Agent foundations is quite close to part of my research and the call of action for embedded agency was quite motivating
  • I felt like I gained a greater appreciation for why philosophy of the self or of the mind are important in AI Safety and reasoning about models.
Other thoughts1
  • i think it would have been good to have a real technical day on reinforcement learning and how it applies to LLMs in particular (exploration and gradient hacking etc)
Heuristic Arguments and Verification Wed 28 Oct, Aden Power 26 responses · energy 3.9/5
Overall +0.05 as expected n = 22
Content -0.05 as expected n = 22
Clarity +0.05 as expected n = 22
Motivation +0.05 as expected n = 22
Exercises 0.00 as expected n = 17
Readings +0.10 as expected n = 21
Other activities +0.13 as expected n = 8
Pace -0.23 about right n = 22
Difficulty 0.00 about right n = 22
Novelty +0.64 mostly new n = 22
Read what they wrote17
Worked well3
  • I suppose Q&A does not count for this feedback form -- but if it counts, it was really good and enlightening.
  • The talk was good and useful
  • The morning lecture was very helpful!
Would improve10
  • Make some of the lecture less vague
  • the morning talk was pretty ineffective. it was not really clear to me what was trying to be communicated beyond "we should be able to understand the outputs better than simply observing them". talking to Aden and reading helped me to see a bit better what is the ARC research project.
  • nothing
  • today was great, i wouldve liked some more intuition to start the reading, but that is a minor detail
  • I wish the ARC lecture was more explicit if possible.
  • Nice, maybe a bit more help with the actual arguments ARC uses would be helpful. Otherwise interesting!
  • The ARC talk didn't really have any concrete details and the presentation could have gone into the mathematics in more details. The rest of the day was mostly self reading.
  • The lecture in the morning on ARC overview and the exercise in the evening were useful. For the lecture I would consider including more real examples - say explaining in detail one of the examples that we later had in the exercise. Maybe this is not possible considering the time limitation. Overall there are no complaints.
  • Not entirely clear on the specifics of this research program.
  • Not much. I think the ARC research agenda is interesting and I was glad to learn more about it.
Changed my mind3
  • I really do not understand how ARC thinks that ambitious mec interp is going to work in general. Even very simple models are very very very hard to analyse.
  • Even though interpretability was not explicitly mentioned, the framing given by the ARC speaker makes me believe more than it would be useful. If we can use the interpretation of a part of the network to simplify part of the task of predicting the output, we are one step closer to understanding, and half a step closer to tuning/controlling for alignment.
  • Was glad to get an idea of what they were doing at ARC.
Other thoughts1
  • I think the 'worked better than i expected' style question is functionally useless given that you have no prior on my expectations. a simple 'how did you find x' on a 1-5 scale would probably be more informative en masse.
SAEs and Activation Geometry Thu 29 Oct, Eric Michaud 25 responses · energy 4.1/5
Overall +0.63 better than expected n = 24
Content +0.54 better than expected n = 24
Clarity +0.46 better than expected n = 24
Motivation +0.54 better than expected n = 24
Exercises 0.00 as expected n = 9
Readings -0.17 as expected n = 12
Other activities +0.13 as expected n = 8
Pace -0.04 about right n = 24
Difficulty 0.00 about right n = 24
Novelty +0.38 mostly new n = 24
Read what they wrote21
Worked well7
  • Eric's presentation was very well prepared.
  • The guest lecturers
  • interesting topics and well explained
  • I had a lot of common interest and similar research taste as the teachers, I enjoyed a lot the lessons.
  • The supplemental readings were a nice touch
  • The slides had lot of concrete examples of what was being described, something that was missing on the other days.
  • Excellent speakers.
Would improve8
  • Both lectures were well structures and well presented. I personally liked both sessions because several emperical results were discussed. For future iliad I would dedicate one day for each lecture with more exercises to practive. This might make physics people more happy :)
  • Some exercise sessions would have helped.
  • Maybe some more exercises
  • very good day nothing
  • nothing to improve, all very good !
  • I would have included this module much earlier in the course as it provided some insight into how to explore activations in a structured manner. It was only implied/inferred from the material on other days.
  • Nothing to improve today. The presentation was great -- lots of nice pictures and well-explained what the concepts we were discussing exactly were!
  • Great talks! Not that much interaction, though. I feel like having a moment to discuss some topics with a neighbour, or a moment to do a quick quiz or something like that, would be a good activating intermediate activity.
Changed my mind6
  • Sparse auto encoders seem less useful than I expected.
  • nothing really just learned a bunch of stuff
  • I migth have moved the work today to earlier in the month, so that it was shortly after covering SAEs
  • Not much change in opinion, had already seen much of the material before
  • I didn't particularly change my mind. I was already interested in the possibility of concepts being represented on submanifolds of the activation space, but my interest was reaffirmed and heightened after seeing this presentation.
  • I did not realise some people consider today's topic as the most (only?) important set of problems. I updated my priors accordingly.
Project proposal day Fri 30 Oct, Leonard Bereska 4 responses · energy 4.3/5
Overall 0.00 as expected
Content 0.00 as expected
Clarity 0.00 as expected
Motivation 0.00 as expected
Pace 0.00 about right
Difficulty 0.00 about right
Novelty +0.50 mostly new
Read what they wrote4
Worked well1
  • David's presentation on the AI Safety funding landscape was helpful.
Would improve1
  • Some feedback on the proposals would be nice.
Changed my mind1
  • There seem to be more promising directions in data attribution than I previously thought.
Other thoughts1
  • A fifth week on doing a research sprint can be pretty interesting!

Reading the bars

Every bar is the answers themselves, laid on one axis with the middle of the scale at the centre line. Left of centre is worse, right of centre better, and the widths are comparable from row to row.

much below below as expected above much above

Pace and difficulty are a direction, not a quality: the centre is the target and both ends are a drift away from it.

on target one step off two steps off

A row says n only where fewer people answered it than answered the day, such as a question about readings on a day that set none.