Pick your name
Aden PowerArtemy KolchinskyAshe Vazquez NuñezBrianna Grado-WhiteCheryne JonayChristian CopettiDan WilhelmDavid PrinzDavid QuarelDavid ZeidlerEdmund LauFernando RosasGeorge RobinsonGuillaume CorlouerJames SykesJulian SchulzKai OgdenKyriakos KatsamaktsisLeon LangLeonard BereskaLeonardo Costa LesageMargot StakenborgMatthew Farrugia-RobertsMiguel Donaire Arcas-SariotPierre PeignéSatya BensonSimon SchraderStephan WäldchenXavier PonciniZach Furman

By cohort

August 2026

November 2026October 2026September 2026August 2026June 2026April 2026

Rated against what each person expected. 0 is as expected.

AI Alignment Mon 10 Aug, Leon Lang 33 responses · energy 3.9/5
Overall +0.03 as expected
Content +0.12 as expected
Clarity +0.52 better than expected
Motivation +0.61 better than expected
Exercises 0.00 as expected n = 8
Readings +0.03 as expected n = 32
Other activities +0.21 as expected n = 19
Pace -0.36 too slow
Difficulty -0.55 too easy
Novelty -0.33 mostly review
Read what they wrote72
Worked well17
  • Loved the discussion and sharing ideas.
  • I loved the group discussions :)
  • Discussion sessions and socialising exercises
  • The readings were really helpful and the questions were good too. They needed some form of thought provocation
  • met others, learned more about AI safety philosophy
  • The vibe and the plan on what to learn
  • Getting to know everyone well first
  • Discussions
  • The main lecture by Leon. I don’t think we can start the intensive without an overview like this one. At the same time, it feels pretty hard to deliver a lecture like this one in any finite amount of time.
  • I did enjoy talking to other people and felt like this facilitated well!
  • I think the introduction part was long but good! Feel I got to meet the bulk of everyone and it made the discussions way less stressful.
  • Food was super great, there were similar sandwiches in iliad 2025, and those seemed the same, they’re really great Some of the readings I didn’t know before were nice, I wasn’t super familiar with CEV and now have at least a hunch (although this is something I would like having more time to actually read the paper)
  • The lecture and the reading groups!
  • The first half-day of just socialisation was very good. It was especially effective because of the rules to ensure people talked to many people in different formats instead of the same people over and over. It might have been better to give the meetings more time (say, 10 minutes each).
  • The level was exactly right for me
  • The discussion was helpful. We split the reading between the four of us and trade readings and summaries. Everyone describe the reading well as they are good explainers.
  • The lectures were actually very clear and easy to follow. Also the socialisation activity at the start of the day was really helpful.
Would improve27
  • getting people to form teams for certain tasks could be improved
  • Defining terms a bit more clearly
  • I felt like some of the reading and material was a bit vague, e.g the volition stuff.
  • Pretty satisfied with the material
  • The 1on1s were very intense. It would be nice to have some break inbetween; at one point I completely lost track of who is who and who I have talked to already
  • We spent too much time on the presentation
  • It would be nice to go faster/in a bit more depth about what’s close to the frontier in terms of research.
  • Things were ok but a little disorganised. Eg we went over time in the lecture and the discussion part felt quite rushed
  • A bigger hall maybe. And also the first session is a bit heavy on the philosophical side of things.
  • Less time spent just talking in the beginning, maybe some game or something? Less computer problems during the presentation.
  • I think the slides could have a more interesting narrative. They felt kind of piecemeal.
  • A more diverse set of activities to meet people beyond 1-on-1s. Maybe games, questions to answer, etc.
  • Dont make this a competition by telling people they get evaluated. Creates weird atmosphere. Also biases towards extrovert personality types.
  • too many iterations meeting people, four sets of readings and discussions was also too many iterations. would be better to rotate groups or ask to do readings beforehand
  • A bit less timewasting.
  • Have mentors occasionally supervise the discussions.
  • Organise a little bit better the 1-1 time so that there is more time for the discussion of the readings.
  • I feel like we got bogged down a bit in the slides and the 1-1 in the beginning seemed a bit long
  • For learning, I would consider making it more clear which things are definitions and which are more philosophical concepts. I understand the goal is to touch on both but some of the confusion was not being able to distinguish the two. Overall I think it was a great introduction and I found it very helpful!! :)
  • Group discussions are nice but it was a bit of blind leading the blind. Also 2.5 hrs for intros felt a little long. I think a bit more explicit stress in the lecture (like 5 mins) on why safety is actually hard (like why control is hard) would have been good for me to hear.
  • Readings were super cool, but we didn’t have time to read them. Discussions were not super productive, some people from my group didn’t seem to understand the concepts (for ex., inner misalignment needs some diggesting, concepts are mostly philosophical, so if you don’t take time to digest them, they will seem useless or not worth it or not well defined- which is what people from my group thought. It was hard trying to convince them this concepts are actually temptative ways to make our intuitious about alignment a little more rigorous in a more philosophical/conceptual way) Some improvement ideas: actually have a little more time to read (like the 20min suggested on the website, because since class was too long we only had 2h for the 4 sessions), maybe send this to us beforehand and say the readings are mandatory previous work (then if people don’t read that’s on them), have less time for the discussions maybe? I think since this iliad is already short, most of the more conceptual readings could’ve been homework and we could’ve done more cool stuff (for ex., RL day was cut short and I’d say it seems better than today?) Unrelated but the class is a little too full and with people coming in and out all the time it’s kinda noisy/bothering
  • It might be too early for this and probably it will come later on, but a mix of non-technical and technical readings would potentially be fun! It would also be helpful (but maybe unrealistic at this point) to have some more precise technical questions to think about in the background, while we learn the material.
  • The topics covered. Most of them felt like arbitrary categorisations - "definition tennis", so to speak. For example, today did not at all make clear why it matters whether a misbehaviour by a model is "reward gaming" or "goal misgeneralisation" - so why spend so much time learning about these definitions? It's possible that the distinction is quite relevant in certain kinds of AI safety research (for example, there may be an intuition that a certain AI safety measure can combat reward gaming but has no chance of preventing goal misgeneralisation), but then it would make for much better motivation to discuss the difference later when a motivating example is available. I feel this was also reflected in the questions and discussions of participants today. Most points felt like they were just about unimportant technicalities of the definitions - not through any fault of the participants, but because they were trying their best to take the content seriously and getting confused (as did I) about these definitions which are really attempts to cluster something with a fuzzy boundary.
  • Give more time for reading sessions, or focus the reading material more on the important/relevant ones; it felt a bit rushed, and I would have liked to dig a bit deeper into a selection of them
  • I strongly feel like the material should briefly cover multi-polar failures and pluralistic alignment. In my limited but somewhat informed opinion, this area of research is quite neglected and at least it is important to make sure that people who chose to work in alignment is at least aware of this failure mode.
  • I think it would've been better to compress today's content into the morning, and do something technical in the afternoon: - Most of us are presumably here because we've already read enough LW posts or similar to be convinced that AI alignment is important. - In the discussion phase, I was often asked to have opinions that I don't think I can intelligently form without having gone through the technical part of the course yet. - Given the compressed calendar, the choice isn't between "present X technical content now vs at a time when people are all present and rested and not jet-lagged; it's between present X technical content now vs not at all."
  • I would probably have structured the discussion a bit more, like maybe asking various groups to pick a topic, but to be honest the freeform discussion worked well
Changed my mind19
  • The importance of fundamental theory within AI research
  • Many different perspectives in alignment that I hadnt thought about much.
  • We did discuss newer forms of alignment beyond constitutional AI eg. ai as willing servants which I thought was fascinating.
  • Too early to say
  • AI Safety, is more philosophical than anything else
  • Little, since most was review of information I already knew.
  • Still deciding between ai safety and math academia. This was interesting info. I found having to explain and defend the intelligence ≠ goal directed post made me understand it much more.
  • No
  • understand more why certain ai safety agendas pursued
  • Alignment theory matters not just practice
  • Earlier in the day I thought we were doing too much social work, immediately changed my mind during lunch and the afternoon discussions
  • I gained language to describe what I’m looking for a bit better: Empirically guided theory work
  • It is too early to say anything life-changing happened but I definitely feel this did not scare me away from AI safety, which was a mild concern going in. The sociability and energy of the whole experience is very good too, could definitely see working with people like this!
  • I don’t think I changed much my mind
  • I learnt about some points that make AI safety more difficult than I would have thought (e.g. even AI system with harmless goals are not necessarily harmless)
  • I read some of the CEV paper by Yudkowsky, which made me think a lot about what "alignment" really means, particularly the "extrapolated" part and the "coherent" part (interpolating different people's values). I am now unsure whether there is really one correct answer to this question, as different humans could have values that are simply incompatible, and especially the concept of extrapolation could have different answers that different people might consider to be the correct answer (for example, I think it's a consistent and plausible opinion to hold that in most cases one should actually not extrapolate and operate by the current opinion only, though I don't personally hold this opinion).
  • The career prospects sounded way more promising than I anticipated
  • I'm more and more concerns about misused more so than misalignment by any parties in the society. I think catastrophic outcomes could come from both human initiate ones more so than misaligned systems.
  • To be honest not really, other than I think I now take the arguments on instrumental convergence a little more seriously than I did before. I very much enjoyed seeing the whole AI safety field laid out broadly rather than just alignment in this initial lecture
Other thoughts9
  • Wouldve been nice to have some more time to read.
  • No
  • Some feedback on discussions and exercises would help.
  • The readings were a bit too much to handle for the time allocated
  • Overall good a looking forward to the program. Responded were a bit negative due to not getting through much today.
  • It might be nice to have some socialisation with other people at Lighthaven or otherwise connected to the topics of the Intensive (of course, only if they are interested in or willing for it).
  • The presentation could have been a bit faster and it would have been beneficial to cut the emerging discussions a bit shorter, as this wasted quite some time with limited benefit for the whole cohort Also, a lot of time was spent on seemingly older concepts, which might be a bit dated? It would have been beneficial to make it more explicit what is actually still relevant today and what is currently the state of the art …
  • I think Leon did a great job of summarizing and explaining the very vague terms into graspable concepts with useful examples.
  • N/A
Singular Learning Theory Tue 11 Aug, Zach Furman 31 responses · energy 3.9/5
Overall -0.10 as expected
Content +0.48 better than expected
Clarity -0.10 as expected
Motivation -0.19 as expected
Exercises +0.52 better than expected
Readings +0.75 better than expected n = 12
Other activities -0.50 below expected n = 4
Pace -0.03 about right
Difficulty +0.39 too hard
Novelty +1.13 mostly new
Read what they wrote59
Worked well15
  • The exercises and technical aspects in the talk were nicely explained
  • The exercises were amazing way to actually reinforce understanding of the topic
  • Loved the handson exercise solving
  • Desmos graphs were great!
  • Zach's high level break down of all concepts without scaring people off with heavy duty AG lingo.
  • Exercises helped a lot.
  • the lecture!
  • I think doing exercises was important, albeit unpleasant at times.
  • Doing exercises to clarify concepts
  • First half of the slides
  • slides were a good intro. many problems were doable at least until the end, the tutorial as a whole is likely a good resource if we go start to finish rather than skip around
  • - The notes are amazing
  • Discussing the exercises with peers.
  • The morning lecture at the beginning was useful. I also appreciate that Zach talk about his own interpretation vs some consensus in the community.
  • The excercises forcing us to put what we learnt into practice was very useful, it's nice to go home feeling like I've actually pressure tested my understanding of it
Would improve23
  • I would have liked more references to follow up on some of the math intuitions.
  • Just one sentence on big picture regarding ai safety!
  • Nothing really.
  • I’d like more time to process and work through the exercises as well as perhaps more time going in depth. I realize that this reads like two things pulling in opposite directions- I guess I was most interested in SLT coming in and would’ve liked to spend more time on it! But definitely came away with a clearer understanding and a vague idea of where to go from here!
  • We talked about it in the end but it felt unmotivated, partially due to scehduling. But I feel that I understand today but am not compelled by its application to AI safety.
  • I would make it a bit clearer exactly what exercise the class is working on currently, especially since you are encouraged to go and work outside for a while (I also missed some 'lecturing' parts because I didn't know we had been summoned back in).
  • Slt could have two days dedicated to it with lots of exercises to motivate the material. Also, showing the connection to safety would be useful.
  • During the morning session, the questions round extended too much in time and it was hard to engage.
  • Maybe spending a bit more time explaining how SLT fits into AI safety (there's a chance this was included and I missed it) Nit-picky, but maybe alternating more often between lecture and exercises
  • Lecturers spent to much time answering irrelevant/tangential questions. In general, I think the presentation could have been *half* as long as it actually ended up being if people stopped asking all types of irrelevant questions. I don't blame people for being curious, but I really would wish that lecturers more often say "you can google/claude this" or "come to me during the break". I also think the presentation was not that good and it was not always easy to follow (from the exercises/texts things were easier).
  • Maybe going over some exercises together early on, as some of the definitions felt different to how we compute them in practice. I think it was great to have solutions, but would have been helpful to maybe work in groups? I worked with a buddy, but talking to other people it seemed they felt like they were the only ones struggling. Also the timing for the exercises seemed fast (as in, too little time to work) but once we see the relevant trick in the solution, the short duration made more sense...not sure how to improve this hahaha, maybe
  • When explaining what happens in the bayes case, you could make it very clear on the slides what the type of each object is, and maybe also explain what the bayes setup is. Also maybe explaining/excluding susceptibilities.
  • Sometimes the lecture explanation felt rushed and confusing, eg some equations or plots shown without properly defining variables etc. Also connection to safety a bit unclear
  • Let speakers know that they can skip answering questions for the sake of time. We should have done the mysteries of deep learning today. Feedback for Zach in particular: I got the feeling he kept getting nerd-sniped by the questions. I think the slides could be organized in a more focused way.
  • more organization around solving the problems. probably want to split into groups and do entire sections at a time rather than 5 mins for a somewhat involved problem. also if a problem requires extra knowledge (even simple such as Taylor expansion or knowledge about what invertible implies) may want to add it to the hint
  • - The written material is amazing, really well written. Nothing needs to change here I guess, except maybe making more materials like this for more complex stuff, but it does is super great to follow, exercises broken down into easy parts that allows you to construct the theory alone, super fun - The class took more time than it should and thus we had less exercises time, and did less exercises than what was initially planned. Also, we had to rush through a lot of them. In the first lecture of the day, people were doing 10 "curiosity" questions per slide (questions that are not like "I don't understand the material, please re explain" - which I'd say are fair and fine, but the questions were actually like "I saw this kind of behaviour how does this relates", which is also fine if not done in a way that actually prejudicates the schedule of the day). One easy way to improve would be to say for the professor to be more mindful (for ex., when prompted with a large "curiosity" question the professor could provide a 10second brief answer and say "ask me later" or something)
  • I would not do a session of timed exercises. Instead guided sessions or everyone works in their own pace.
  • A lot fewer questions during the lecture. I didn’t get much intuition out of the lecture, it would be nice with a concrete example. I did the exercises and often thought afterwards: why do I care about this? How does it relate to safety? Sitting inside the lecture room and doing exercises was uncomfortable and didn’t allow for discussion with peers. But when I left to do the exercises on my own I didn’t know when the lecturers would go through some material. Would be better if they let us work on the exercises in groups and gonged us in every 25 minutes or something.
  • For me, some basic concepts were explained a bit too exhaustive/lengthy, while being quite fast on the more interesting/involved topics; again, I would have preferred to spend less time answering (partly only tangentially related) questions, and stick more strictly to the planned curriculum — this way, we would not have needed to skip the more interesting advanced parts …
  • Talked to Miguel the TA and apparently many of the prerequisites and the build up got cut quite significantly. I'd prefer to do the Mysteries of Deep Learning first to build up the intuition before we go into the materials today. I also wish that we could be instructed in advance what sections and subsections to read prior and do the preparation of the reading prior to the class. The lecture in the morning where we talk about intuition was quite helpful. At this point, I'm still uncertain as to why and how degeneracy is relevant to generalization. And most importantly, I'm still unconvinced as to why SLT at all apart from that they are mathematically beautiful.
  • I think any course content should either be i) taught during the lectures as if we're completely unfamiliar with it, or ii) something we're explicitly told well in advance to learn before the lecture. The first hour or so of the morning session was great, but then suddenly it got way more technical and explicitly assumed learning theory background that I didn't have and I got completely lost. I was doing okay in the afternoon sessions, but then started to fall behind when the exercises started referencing content from Section 1 that we had skipped. At some point, once I realized we'd be largely going through the material independently during the session anyway, I decided to just proceed linearly from the beginning of Section 1 and ignore the instructor's pacing. Then during the break in the middle of the afternoon session I was having an interesting/productive conversation about AI alignment with some of the other attendees, so I opted to continue that conversation for another hour and a half rather than return to the session. I plan to return to the material this weekend, after the Mysteries of Deep Learning lecture.
  • The pace was a little bit irregular, and I would have appreciated spending more time on the free energy equation and deriving it, rather than all the other things which could have perhaps been given as initial readings.
  • honestly I thought today was pretty much fine. The lecture was really good and the excercises were quite helpful. If I had to change one thing it would probably be doing fewer, but harder excercises, rather than many short ones, however, I definitely felt like the questions forced you to use conceptual understanding rather than just turn the handle which was good.
Changed my mind15
  • Want to learn more about slt
  • Great to learn about deep theoretical frameworks to analyze deep learning models.
  • further validated my gut instinct that SLT is both beautiful mathematically and useful empirically
  • Thought that analyzing loss landscape was hopeless, but SLT changed my mind. It is a worthy attempt at generalization.
  • Learning Bayesian stuff might be worth it (which means a lot coming from an algebraist)
  • SLT seemed different from what I expected.
  • I didn't know what SLT was about and now I kind of do.
  • SLT is interesting. I heard people describe it as rather “esoteric”, but there are some nice concrete results
  • I am leaning more towards switching to a career in safety now.
  • SLT seems more approachable than before, was able to do more exercises than expected
  • Understood a little more details about SLT, great
  • SLT seems less clearly related to safety than I originally thought.
  • I'm uncertain on why SLT at all and why SLT is popular in alignment + safety space only but not elsewhere.
  • I was new on SLT, and the free energy equation and decomposition were very interesting.
  • Not really, I came into this sort of knowing that SLT was difficult to apply in practice to things like language models, and also that it helped explain some cool results, however it was very nice being able to finally take a good look at them
Other thoughts6
  • Again just a sentence or two on relation to safety. I can imagine but would like that confirmed or rejected
  • It would be nice to moderate questions from the participants somehow. It seems some people are trying to make themselves seen for the fellowship and the constant questions make it super hard to concentrate onto the actual lecture; which then tends to run late
  • We should limit the amount of questions per person
  • A lot of unnecessary and rather trivial questions were asked today, which made the thread quite difficult to follow. I also think the structure of the SLT talk could be improved. We didn’t manage to cover many of the important details, which left some gaps in understanding. This wasn’t because of the teacher, in fact, I think they did a great job, but mostly because the large volume of questions disrupted the flow and took up a significant amount of time. The exercises also didn’t help as much as they could have. I think it would have been more effective either to work through them in a guided way or to allow more time, rather than just 10 minutes per exercise.
  • If I were to get into SLT more, I'll probably be reading the grey and the green book first and catching up on the materials independently first then review the lecture.
  • N/A
Computational Mechanics Wed 12 Aug, Charles Renshaw-Whitman 30 responses · energy 3.9/5
Overall +0.27 better than expected
Content +0.27 better than expected
Clarity -0.13 as expected
Motivation -0.03 as expected
Exercises +0.17 as expected n = 29
Readings -0.20 as expected n = 10
Other activities +0.11 as expected n = 9
Pace -0.03 about right
Difficulty +0.37 too hard
Novelty +1.00 mostly new
Read what they wrote41
Worked well11
  • Most things, the anarchistic format, the short lectures, the announcement.
  • How this fits into the broader perspective of instrumental convergence of goals and how this can act as good deferral
  • Exercises were great
  • The exercises.
  • Liked speaking with the instructors who had quite a lot of good intuition!
  • - the lecture itself - alternating often between lectures and exercises
  • People didn't do dumb questions out of context or only for "curiosity" mostly, amazing. Actually maybe today I was the one doing this.
  • Talking to Ben & Charles was great, very knowledgeable guys
  • interesting and relevant topics I wouldn't have thought to include myself
  • Exercises were very instructive once understood.
  • I liked the excercise which was a guided discovery of the simplex results
Would improve16
  • It would be nice to connect to control theory and predictive control and tell how they are similar and differences to comp mechanic ala "many faces of RL" in David Silver lecture from 2015. Today lecture and format was awesome. And I like the announcement in the morning to know what to expect. The explanations were also useful.
  • Specifically today I think everyone would have benefitted from slightly more lecturing time. Even though the notebooks were well designed it was kind of unclear why we care at some points and notation use was sloppy all over the place.
  • It seems very instrumental to the alignment problem, but the material seems very dense and difficult given that it is fairly new
  • I think the exercises could have been better designed (I found it a bit difficult to parse it in the time allotted)
  • comments on "big picture", what is the field currently doing and how it connects to safety
  • I am not sure what to improve despite thinking it can be improved. I think the instructors did a great job and answered/ motivated the material well.
  • I think the day could have a lot more exposition in the notebooks on Markov chains and the transformer section; the exercises were honestly not very difficult and the most confusing part was understanding the notation definitions which were not very clear. I essentially only was able to complete them because I remembered the topics pretty well but had I not it'd have been pretty difficult to parse what the questions were asking.
  • very subjective, but maybe spending a bit more time on the big picture?
  • As I said in yesterday's feedback form: _please_ tell us in advance what background material will be relevant for a given lecture, so we can learn it in advance. I had a hard time answering the above multiple choice questions, since for all I know, today's content was actually perfectly calibrated to people who already had the requisite background. I'm already familiar with Markov chains, so the morning content was straightforward for me (too easy in fact), but then I got completely lost as soon as it assumed deep learning background.
  • (small) Suggestion : Turn the notebooks 1 and 3 into a set of lecture notes like the SLT notes written by Kai, improving on the theory, constructing all the details as exercises, this would be VERY awesome (but this is a small suggestion, I generally think it's fine as it is)
  • Make it very clear at the start how this connects to LLMs and AI safety, and what the goal of this technique is. I think the measure theory part made people more confused on things that were already presented.
  • I think the problem with the exercises was not so much that they were difficult, but that there was that there was *a lot* of notation being introduced. Also, I often didn't really know what an exercise was asking of me - so I think there is a lot of potential to improve the exercises to make it easier what is being asked
  • exercise math notation could be improved, make notebook 2 more accessible if didn't fully finish notebook 1
  • HMMs should have been an initial reading and I would have started faster with the deep learning experiments etc.
  • Connection to applications. More carefully written exercise notebooks - these were generated by Claude (which is fine), with a typical habit of LLMs without prompting to use jargon and vibe-y language that made it quite harder to follow. Clearer explanation of Hankel matrices: turning finite-rank Hankel matrices into GHMMs, and how this works when it is just approximately finite-rank. This could have been better as a lecture/talk/presentation with exercises to reinforce, rather than just exercises.
  • I reckon spending a tiny bit longer on the lectures rather than the code would have been helpful but only a tiny bit.
Changed my mind11
  • I didn't know that comp mechanic is basically predictive control!
  • This does seem progressing
  • I did not know much about computational mechanics and now it is my favourite research direction
  • This belief state geometry approach seems promising
  • Belief state geometry was more interesting than I expected
  • leaning further toward empirical research honestly/learning more concrete physics theory
  • I still believe comp mech is a cool theory, I'm still not super sure Simplex work applies for more than toy models and I still believe comp mech for world models is an awesome direction (that I'd be happy to pursuit XD)
  • I have a bit more faith in Simplex's agenda than before even though I was already pretty pro-Simplex before
  • yes world models and yanns focus on them make much more sense now
  • Added support for understanding ML algorithms as an approach
  • I had quite a few ideas on possible follow up experiments using computational mechanics that I want to try out so I genuinely left feeling inspired by it. Also I am going to look into Hankel matrices at somepoint.
Other thoughts3
  • I feel like some mathematicians in the room are too focus on trivial details that aren't that important to the main point like the negative infinity question was borderline annoying.
  • Context on the feedback: I read "Computational mechanics: pattern and prediction" and "Computational mechanics of input output..." papers from Crutchfield before, I had some context on the Simplex agenda as well (from "transformers represent belief state geometry..." on lesswrong, so of course I know how this fits, but I think people seemed confused about how this relates to safety. Even having read the papers, this was helpful for me because I was always confused on the "bayesian belief update and dynamics happening at the same time" thing and doing the math by hand on the notebooks was super awesome. I also had not much understanding on the GHMM stuff (people be like "negative probabilities how awesome"), so this was helpful. Also notebook 3 was kind of super awesome (approximating epsilon machines? the missing piece from crutchfield papers that I read).
  • N/A
World Modeling + Transducers Thu 13 Aug, Fernando Rosas 28 responses · energy 4.0/5
Overall +0.75 better than expected
Content +0.79 better than expected
Clarity +0.71 better than expected
Motivation +0.36 better than expected
Exercises +0.36 better than expected
Readings +1.25 better than expected n = 4
Other activities +1.00 better than expected n = 4
Pace -0.21 about right
Difficulty -0.11 about right
Novelty +0.68 mostly new
Read what they wrote42
Worked well13
  • time for exercises and lectures was good
  • The structure of the course: i.e how long the talk was and the length of exercises. Things fit together nicely.
  • I liked starting with the motivation and then getting into the details.
  • The first set of exercises and the second lecture.
  • Both talks (Alec & Fernando) where excellent
  • Loved the balance of theory and coding exercises.
  • Now I can finally read the fractals in Simplex's papers!
  • Alec and Fernando are great teachers!
  • Exercises worked great. All Lectures were at an appropriate pace.
  • Alec was very good.
  • The first class in the afternoon was the best we had until now, despite problems
  • The world models introduction was helpful and fun, the transducers section from the second part was very helpful, and the sneak peek of Alec's research results was very interesting
  • I think the excercises were actualy really helpful
Would improve17
  • MSP was not defined in the lecture notes, but it was what we were asked to draw in the exercises. Might be good to do a similar example before the exercises.
  • Not much. Perhaps more discussion about the relation to the previous day.
  • I would like a bit more open-ended research-style questions. Those motivate me more than computations. Also there could be a bit more said about probing world models as I think that is quite important.
  • I think the first lecture should have more content. The second set of exercises were not very helpful-there was strong overlap with yesterday's Jupyter notebooks and also were super computational (in an unenlightening way). Finally, I thought of a suggestion for yesterday-have the lectures follow the notebook and connect the notebook's terms to the lecture/AI safety. I found the exercises to be very tractable mathematically, but not connected to AI safety.
  • The exercises could have been a bit more interesting I think - a bit more mathy and/or a bit more conceptual
  • Slightly more challenging exercises.
  • One of the best teaching and learning experience this week. I like the high level overview in the morning and the more hands on approach during the day.
  • yeah I think the work was insane. It only makes sense why alignment is incredibly difficult and a theoretic perspective is even harder
  • I think it was great to see a lot of the research in practice and how the objects we learned about in the morning was relevant to results we were showed
  • Use the gong more often, I was late to the lecture because I was waiting for the gong.
  • More time for exercises is always better. Maybe useless critique because I'd guess for most people this was not a problem: I like to be very detailed with my probability (for ex., use Kolmogorov extension theorem to actually make sure giving prob of any word suffices to describe uniquely prob over bi infinite sequences), so the class was great in the temptative formalization of stuff, probably the best we had until now BUT the way physicists/not mathematicians handle this transducer thing is very confusing, of course if you draw an HMM it's easy to compute stuff, but the foundations from the class were a little messy. For example, we define an interface anticipation free iff p(y_{:t}|x_{0:})= p(y_{:t}|x_{0:t}) and this is confusing because then if I define my process to be Y_t = X_{t+1}, how do I compute p(y_{:t}|x_{0:t})? Answer: you actually can't unless you put a prior over inputs, but this notation of using "conditional symbol" when in practice we're actually having a markov kernel and then anticipation free means something a little bit different was kind of confusing. Not too big of a deal for most people I guess, I just solved this problem asking claude to formalize every little piece for me so things became clear, but I do wonder (and this is also the case for the other days, less perceptible for me because I had more context): if the plan of ILIAD is to bring more mathematicians into alignment through the intensive, then perhaps the classes should be more careful to not be hand wavyy, this keeps applied cs, applied physicists, etc. happy but myself - that am not even a complete mathematician - got a little sad with the cumulative of hand wavyyness with stuff, maybe other people feel similar (to be clear, there is a bunch more relevant stuff than my tiny prob detail that is hand wavyy and not precise, it's just that I'm being able to solve the formalizations through previous contexts + used to translate physicists/other areas stuff to actual math, but at some point I won't have this context). To be clear I don't mean that I want everything super formalized and complete, precisely because this is sort of impossible in a new field, I'm used to working on under defined problems for example, but I think doing the work of separating exactly what is under defined and what is clear and communicating is kind of important and somewhat the major benefit we get from classes - structured learning as opposed to reading papers (think of someone using the unknown hypothesis/theorems as black boxes, but they're isolated)
  • I think we spent too much time on the concept of world models in deep learning which even according to the presenter does not have an agreed upon definition. Also, the afternoon content seemed pretty similar to the content from the previous day but with new notation etc. While new to me I left without knowing why it was needed vs the HMMs from the earlier day.
  • I would love to see what this framework was being used for!
  • great lecture! Maybe it would be potentially helpful to include more open questions?
  • I was already comfortable with automata / Markov chains / Bayes' rule. For me, the entire challenge of the exercises was in parsing the notation, and getting code to run. (This doesn't necessarily mean the curriculum needs to be changed though, since other people don't have my background.) As I think I've said every day so far, I wish the connection between the math and AI alignment would be more explicitly explained.
  • This (transducers) seemed very redundant given yesterday (HMMs). It's just HMMs where you can switch the HMM transition probabilities used each step based on the input. The concepts and the exercises could have been gotten through much faster.
  • Today was a really good session. The one thing that would have been more helpful would have been a slide with all the definitions introduced in bold just before the excercises but that's it.
Changed my mind7
  • I can see the hype about world models.
  • I think this is a very interesting direction.
  • No change
  • Still kind of unclear of what the world model is supposed to be. Seems like there is a big gap between the theory and the toy examples to real life problems. Would love to investigate how this scales.
  • Some of the works are quite similar to my on-going work in Multi-agent RL. Looking for specific ways to connect the language.
  • Amazing that these toy models look very promising
  • I came out finally understanding belief state geometry fully, and it cleared up a few misunderstandings about it that I had absorbed so that was really helpful. I also came out with some more ideas of computational experiments I want to try out now.
Other thoughts5
  • would be useful to make sure the gong is always sounded before a session starts
  • Fyi - I will be missing a few hours tomorrow due to a medical situation. Nothing I need help with, not an emergency, but something that shouldn't wait
  • Great that today there were group exercises.
  • Sorry, I forgot to submit this yesterday!
  • N/A
Mysteries of Deep Learning Fri 14 Aug, Zach Furman 32 responses · energy 3.9/5
Overall +0.41 better than expected
Content +0.50 better than expected
Clarity +0.59 better than expected
Motivation +0.81 better than expected
Exercises +0.33 better than expected n = 3
Readings +0.38 better than expected
Other activities +0.50 better than expected n = 6
Pace -0.19 about right
Difficulty -0.22 about right
Novelty +0.19 an even mix
Read what they wrote42
Worked well9
  • I appreciate when Zach just tells us his genuine opinions because it is indeed the most useful parts beyond whatever reading materials. I wish that the whole day is just pure genuine opinions even though it's niche because it would make me be able to assess and think for myself better.
  • the whole premise of trying to understand why scientific understanding of deep learning is ACTUALLY hard and how empirical developments led to a complete paradigm shift in learning theory. working through the relevant concepts like (un)learnability and expressivity and the gap between them, what kind of loss landscape geometry you have around hard to learn functions etc was particularly instructive
  • Forcing group discussions worked!
  • Lectures
  • some interesting topics, interesting additional readings
  • Loved the discussion
  • I think the discussion format was well-suited for the task at hand.
  • The program synthesis article has a very interesting hypothesis that I wasn't aware of, and the approximation article was a very helpful read.
  • The readings followed by discussion worked quite well together
Would improve20
  • I didnt like that what youre left with in the class is like "yeah deep learning is weird, dont be fooled to believe statistical learning theory works or whatever", but the experiments that show that cam be explained quickly, i feel most of the class shoulve focused instead on explaining the approaches to explain these problems (and how they fail)
  • I think Eigen hall is a little too cramped for all of us. It would be best to use Cantor hall generally. I think Zach is a good writer and a good curator of relevant resources.
  • Don't overcomplicate and spend too long on a similar theory
  • today worked really well actually i can't
  • A lot of "qualitative" content and discussion that was kind of just opinion-based, rather than something technical. Many of the readings felt like rehashes of the first one, or of things said on previous days. Moreover, the discussion felt somewhat but not very novel, and it seemed like we often moved on before discussing things to the satisfaction of those who raised them. I think it would help to use discussion formats with some kind of break-out allowing small groups to discuss and debate in detail (and this needs to happen _after_ a round of reading and sharing takes or questions with the whole group, as this generates potential discussions).
  • this was a discussion about the broad landscape of research. I wish it had focussed more on safety prospects and not only on the science aspect
  • Scheduling it first would be better.
  • I think today was too vague; I think it is hard to internalize or learn things without doing something concrete. I get that this was more of a high level meta overview day, and maybe it is just hard to feel the learning that is happening, but I fear that I have learned to say something that sounds correct without properly grokking it.
  • Perhaps more freedom on what to read? As in "pick yourself what to want to read, XXX will be discussed from "10:10 -> 10:20", YYY from "10:20 -> 10:30" etc.
  • This would make more sense earlier in the schedule, but I assume you already know this
  • It felt like a lot of repetition. Maybe it would be good to make this 2/3 or 1/2 of a day, and do less discussions/cut even more of the readings we actually do today. The program synthesis blogpost felt like a repetition of slides+things in other papers.
  • crash course on statistical learning probably; the way that it seemed like computational mechanics was a crash course prior to world models and transducers
  • Have more mathematical exercises in groups
  • Some of the discussion questions were too easy and uninteresting to talk about.
  • imo the best format is to have us disperse from the lecture hall, form groups, and do a few readings and discussions at once. having us all in eigen continuously is cramped. it's difficult to form groups, and many people closeby were not interested in discussing
  • I was quite familiar with the content. I would love to have another chance to dive deeper into some theory like SLT
  • Better than I expected, mostly because I expect these conceptual days to be kind of boring, but the blog posts were actually great, didn't read most of them previously, nice. Given this context, the part we read 5 pages of a book without context/access to some definitions in the book was kind of hand wavyy, so maybe write a small blogpost with that content contextualized and use it - but this is somewhat minor. The PRH and induction heads paper are kind of bad tho (PRH paper is methodological crap and induction heads is in mech interp), so maybe some other readings would be better. Also, looking at the April docs curriculum, it is perceptible that the ordering of conceptual days is made in a way that systematizes conceptual stuff, which seems very cool. This day for example systematizes well this "approximation/generalization/optimization" intuition/frame, that was cool, but then the ordering got changed and days were deleted so this conceptualization/framing felt a little bit out of place (as in, it seems obvious that you made the curriculum expect it to connect and be in some ordering and it is not and this is kind of bad)
  • Today was great, I really enjoyed it, helped me learn a lot and put a lot of different parts together!
  • I don't know actually for today the reading oriented session felt suited for the topic, and that was executed pretty well so I can't think of anything.
  • Wish this was positioned earlier in the week
Changed my mind9
  • I knew that there isn't much understanding of deep learning theory but I didn't know that it was genuinely so little understanding. I also appreciate that Zach and Paolo have some information about the environmental impact of LLM usage.
  • i now am even more theory pilled than before
  • The whole "Bayesian learning generally converges rapidly" was something I haven't thought about before
  • Platonic representation hypothesis seems to have a strong implication for safety if true
  • I am now at 60% leaning towards grad school (40% AIS). I started at 80%.
  • I'm less skeptical of statistical learning theory for DL, as long as we throw away VC Dim stuff and incorporate data/sgd path in our complexity measures
  • in some sense I understood how mathematics could be relevant for a more overarching view of things (rather than for computational aspects), but also seeing a lack of formalization in the objects involved (e.g. 'space of programs') made me feel like the field is really behind, or doubtful how much progress can be made through mathematics if the starting point feels so far behind the theory that could eventualy help.
  • Why it is difficult to explain why NNs perform very well on approximation & generalisation; how optimisation differs from approximation & generalisation.
  • I feel like I now I have the main surprises of the modern deep learning era in my pocket to explain to people whereas before my understanding was far more nebulous
Other thoughts4
  • Continue group discussions
  • It was a bit hard to read with so much background noise in Eigenhall. Might be a good idea to use the model of Monday wrt discussion (and bring people in every once in a while).
  • I was confused about some practical aspects yesterday, not because of the instructor but because the material revealed gaps in my knowledge (which I mentioned in my feedback yesterday), but actually most of these were confusions were cleared up after discussing with my peers over dinner.
  • N/A
Mechanistic Interpretability Mon 17 Aug, Brianna Grado-White 31 responses · energy 3.8/5
Overall +0.10 as expected
Content +0.19 as expected
Clarity +0.45 better than expected
Motivation +0.55 better than expected
Exercises 0.00 as expected n = 25
Readings +0.48 better than expected n = 25
Other activities +0.25 better than expected n = 4
Pace -0.16 about right
Difficulty -0.26 too easy
Novelty -0.42 mostly review
Read what they wrote44
Worked well13
  • Feel like compared to other days I understood better how the results relate to mechanical intepretability. The TAs were also good at explaining and also visiting each group and filling the gaps.
  • Brianna's willingness to answer all the questions and keep refining explanations until everyone was on the same page. Also shout out to Dan (participant) who has mentored ARENA before, and was able to jump in for adhoc transformer refresher.
  • Readings were chosen well. For people familiar with mechinterp, reading the critiques helped them understand what is missing in mechinterp and what we can hope to get out of it.
  • It was fun to discuss the readings with other students and the instructors
  • interesting readings, handwritten slide formatting nice, able to spend some time in lesson and other time not in the room
  • I ended up reading the posts on "new" ARC agenda that I wasn't aware, that was super cool.
  • It was very useful to understand how sparse autoencoders can deal with superposition.
  • I really appreciate the reading list that Brianna curated. I also genuinely want to hear more of the teachers’ takes because that is usually the best part of the day. I think Brianna did a good job presenting the information. I also really enjoyed the comparisons to modern mechanistic interpretability work. Regardless of my own views on that work, it is generally useful to know about it.
  • Readings.
  • The content included a mix of readings/presentations for people of different backgrounds and interests: - Some content went over transformers, other content went straight to mechinterp. (I'm aware that transformers were technically a prerequisite for the Intensive, but this wasn't well advertised in my opinion, and anyway there's a difference between reading about it once on short notice and actually grokking it. I benefited from this morning's presentation on them.) - Some content was more empirical, some was more theoretical. (I'm instinctually drawn toward theory, so I appreciated the readings on ARC's theoretical mechinterp work.) This made it easy for me to have an interesting and productive day.
  • The exercises, the instructor was very good about answering the material and today's lecture was not overloaded. People had good questions and there was more collective answering, perhaps because the avg student seemed to know more of the material and could jump in to help the more confused individuals.
  • Some of the code exercises were actually manageable and they made me close some percentage of my empirical gaps.
  • I think the mix of coding excercises and lectures was well balanced. I like that a good amount of time was spent actually trying to make sure we understood the content properly
Would improve17
  • Brianna's presentation was good, so this is largely meant as a "content choice criticism". I was pretty bored today, as I knew most of this already (and I guess several people did) - only once we did the readings from ~3 PM did I feel that things got fun (reading e.g. ARCs agenda). I think given how different the background level of different people is, perhaps the best way to do a day like this is "Group 1: People who have use of a lecture; easy exercises", "Group 2: People who know the basics, get reading & exercises", "Group 3: People who are quite advanced, get more advanced readings & open-ended questions". I think this is something that I think would be useful more generally speaking on days where one expects people to have varying degrees of knowledge about a topic
  • I am pretty content with the material. Though, some people seemed to need a refresher on transformers etc which staggered the whole presentation a bit.
  • Some engineering pre-exercises would be super useful! I was surprised some people had trouble understanding what activations/weights were.
  • Having us divide into groups and have do group discussions and pair programming would help.
  • The lectures were bad - skipped over far too much stuff. The readings seemed to not require much technical understanding nor be insightful into mech interp. Maybe it was just that I was tired.
  • Less focus on the transformer architecture, maybe more focus on CNNs since there the architecture is easier and the feature -> circuit -> universality claim is more clear.
  • I struggled the most with today's material, mostly because there were a lot of concepts (eg details of the transformer architecture, or mechanism of a sparse autoencoder) which were more or less new to me and I did not manage to put all the info together fast enough to follow the rest of the class
  • many different experiments and diagrams are on the slides, sometimes difficult to understand without more explanation of the experiments
  • Since this intensive has less days, maybe not putting this day since I'd guess everyone should already know about it (I mean, if someone is wanting to get into AIS and never read even the basics ideas of interp I'd say that's on them and not on the program. You could say this day is actually useful because people code ARENA notebooks but then this was in general not incentivized by the professor, and I also believe you guys weren't expecting people to learn py torch or ML engineering from this day lol). On the other hand, I think this is mostly my beliefs since a bunch of people from the class seemed to be very confused with basics of transformer architecture, so maybe this day was needed - despite very suboptimal. If the day was natural abstractions and/or ARC current "new" agenda it would be so much better, for example.
  • - Nit-picky, but maybe reconfirm the essential prerequisites the night before - Replicating some simple toy models might be helpful?
  • It would be nice for the ILIAD Intensive course overall to start with today’s material, then move to Mysteries of Deep Learning, and then SLT. I think it is also important to begin with an explanation of MLPs and Transformers, regardless of people’s backgrounds, to give everyone the same grounding. I found Dan (the participant who taught ARENA)'s walkthrough of Transformers pretty helpful, and I liked the selection of papers that Brianna curated. Yangdai also gave a really good explanation of the residual stream and Transformers. It might be good to have a manual pencil and paper style walkthrough as exercise as well.
  • Assume knowledge of DL concepts.
  • I feel like it was good to see some of the concrete steps, but I feel like a lot of the concerns with AI safety/ examples of unsafe behaviour sort of happens through more advanced training (RL) or something more involved. To me, this gap makes it harder to be 'inspired' for new ideas relating to safety since it feels like today's material, or maybe MechInterp itself, is further from the research horizon (I could be wrong, but it seemed like this to me). I do not think the material should be cut since it is foundational and it seemed to really clarify a lot of the 'blackbox' for many people, including me, but I wish there was some time dedicated to 'looking ahead' or explaining where the field hopes to go from here. Possibly the reading is supposed to help with this, but I started with the reading titled 'Against Almost Every Theory of Impact of Interpretability', which maybe coloured my perception of MechInterp as an approach.
  • It's challenging to provide honest review here. Today was bad, and it wasn't the instructors fault. It was on us, the participants. It was impossible for the instructors to do anything meaningful today, because we simply don't know what a transformer does. To make things worse, we don't care about the fact that it was a clearly stated prerequisite. Sadly, most of today's readings and exercises couldn't be explored properly as a group.
  • More clarity on background. For example, I spent a long time trying to learn about transformers but in retrospect, I think I could have pretended everything was an MLP without that much loss.
  • I think the main failure mode of running this day without the engineering day before it is that a good chunk of the cohort might lack the relevant PyTorch fluency or general empirical experience to follow or engage at all times. Given the compressed format of this run of the intensive I cannot really say I have any ideas as to how to improve it. In my case, even though I had studied the Toy Models paper, I got a bit lost during some parts of the presentation and definitely just because of lack of empirical hands on experience with SAE-like tools.
  • I feel like maybe a little more emphasis on the problems with SAEs would have been good, so people don’t come away with the impression that they are still like, the standard tool everyone reaches for since they really have fallen out of favour in the field as far as I can tell. However, other than that all the basics were covered really well.
Changed my mind11
  • I forgot how interesting mechinterp was
  • Critiques of mechinterp helped me understand the shortcomings of this field and made me realize that it is ok to question the value and methodology of a field even if the field is popular
  • Nope
  • Interpretability seems to be mainly an architecture thing; so CNNs are interpretable and transformers are less so.
  • Mech interpr is criticised, but it seems narrower definitions of it (eg focussing on a single feature instead of mapping them all) is useful
  • Found out more about open problems in mech interp worth pursuing
  • Apparently, I conflated deep learning theory with mechanistic interpretability. I didn’t realize that mechanistic interpretability is quite narrow.
  • Hardness of mechinterp.
  • I am more convinced that 'steering' and other safety related things (e.g. situational awareness) can be detected by MechInterp
  • At some point I will have to sit down and train a transformer from scratch, me and Claude against the world.
  • Not really I’ve done quite a lot of mech interp already so much of it was fairly standard.
Other thoughts3
  • Another highlight of the day was talking to Yangda about his journey into ILIAD. He connected me with Daniel Chiang, and we have since had many wonderful conversations about agent foundations, as our work overlaps quite a bit. More generally, matching people with researchers working on topics they are interested in early on would probably be a good idea for the next iteration of the Intensive.
  • Second week off to a great start!
  • N/A
Training Dynamics Tue 18 Aug, Guillaume Corlouer 34 responses · energy 3.6/5
Overall +0.68 better than expected
Content +0.82 better than expected
Clarity +0.44 better than expected
Motivation +0.32 better than expected
Exercises +0.30 better than expected n = 33
Readings +0.21 as expected n = 33
Pace +0.12 about right
Difficulty +0.12 about right
Novelty +1.03 mostly new
Read what they wrote39
Worked well11
  • good readings and exercises
  • Everything else. In particular, the slides, exercises, and readings synergized, more so than on other days so far.
  • Lectures were the best so far. The readings were interesting (but hard).
  • Exercises
  • Just the topic was cool.
  • Exercises are good and I like hands-on math. Good mixture of exercises and readings
  • How structured everything was. Having the material in advance, even just 12 hours before the lecture makes a huge difference
  • The material was very accessible and well-explained by the instructor and it was helpful to chat to the TA.
  • The discussion on how depth initialization affects the dynamics
  • The reading where we actually had to write up our thoughts was really effective at getting us to focus the discussion.
  • The mathematical exercises, though pretty easy to solve, were very nice and gave me intuition about what kind of behaviour DLN training is like.
Would improve14
  • a lot of the math is dense if newer to this field
  • The Saxe et al paper is kind of bad, didn't thought it was worth it. The exercises are super great though, I could only do 3/5 because I spent some time talking to Adam from Simplex, so maybe giving more time to finishing the exercises instead of reading this paper would be great (or something else that is not this paper). In general, I think this training dynamics day was one of the most informatives (I didn't have previous context on it)
  • I think the morning slides could've made it more explicit that i) the training algorithm converges to the true solution for this problem, ii) proving this and bounding the rate of convergence here isn't our focus, and instead iii) we care about _how_ it converges for reasons related to alignment, and this will be the focus of most of the slides.
  • I think the exercises could be improved - I think having more text in the statements motivating/explaining the NTK and the connection to saddle points would be nice (such as the explanations in the solution guide).
  • Morning was a bit lecture heavy. Adding exercises in between could slow down the pace a bit.
  • I felt like the lecturer sometimes didnt quite answer the questions.
  • I don't know if the Adam/Paul thing today was planned beforehand, but I would've liked if this had been announced a bit earlier and I think one could have worked around it a bit. I was a bit unsure today if I should focus on talking to Adam or do the exercises (I can only talk to Adam today, but I can always do the exercises. Though *will* I?)
  • I would have loved some more challenging exercises. A lot of the exercises were just one line solutions. They didnt stimulate my brain much.
  • I think the slides have been pretty bad. The slides are just bullet points, but the properties of a dynamical system seem to be visual, so would be nice to have some more focus on intuition.
  • I enjoyed the AMA with the Simplex thing, but it felt hard to return to exercises rather than think more about their research agenda.
  • Make it more structured and clearer channels of communication. Cantor Hall was abandoned from lunch and I didn't really know what I was supposed to be doing.
  • The reading list was well-curated, and I appreciate the teacher posting the prerequisite announcement early with clear communication. The only issue was that the preparation window was quite short. Fortunately, I had already read some of the Saxe lab's work around February for their interview, which allowed me to follow some of the lecture. I like the reading group exercise in the afternoon as well.
  • Today was really good, I don’t really have any critiques.
  • Can't really say much as I was not feeling too well. The one thing that was not so clear I guess is that we were using a very simple toy model (DLNs) while actual frontier models do not use this architecture. I would've appreciated some more stuff on TTNs as the better toy model to study, being both non-trivially expressive and mathematically tractable to investigate.
Changed my mind11
  • This seems an honest shot at understanding NNs. I'd be very up for studying similarly toyish models but increasingly complex with these techniques. More neglected than SLT and comp mech for sure (although comp mech still seems more promising)
  • 55% grad school 45% ai safety. I feel less confident that an understanding of deep learning helps us solve alignment though. I read an argument yesterday that a complete understanding of mech interp would help AGIs improve themself.
  • The math behind dln is both simple and powerful. I feel natural extensions to nonlinear networks should be very fruitful for agi safety.
  • I was not expecting this to be interesting (thought it would be more engineering) but was suprised at how interesting DLNs are. cool!
  • I am really surprised how much you can milk linear models.
  • Today felt like a necessary step of bridging theory and practice in my mind. Theres still work to be done there but today closed the gap a bit, especially after reading the geometry of fibers paepr
  • This seems to be slightly more relevant to safety than I originally thought.
  • I had some good discussion with Zach and found it was helpful to get a sense of what people were thinking about
  • About how difficult it is to move from deep linear NNs to deep non-linear NNs.
  • The clarification that SLT models a static landscape was crucial for understanding how these geometric concepts connect to training dynamics.
  • I had never paid much attention to training dynamics because I had always seen it as something quite opaque, whereas now I see it can actually be analysed quite precisely and in a way that feels much closer to the real world contexts than some of the other things we have learnt about.
Other thoughts3
  • I would really appreciate if someone told people to come to them after the lecture when they ask their third follow-up question that is clearly irrelevant to better understanding what the lecturer is trying to convey
  • N/A
  • I wasn't feeling particularly well and couldn't really make the most of it but I still liked the way it went! I left home early after solving the exercises.
Data Attribution Wed 19 Aug, Louis Jaburi 26 responses · energy 3.4/5
Overall +0.54 better than expected
Content +0.54 better than expected
Clarity +0.19 as expected
Motivation +0.69 better than expected
Exercises +0.28 better than expected n = 25
Readings +0.30 better than expected n = 23
Pace +0.12 about right
Difficulty +0.23 about right
Novelty +1.12 mostly new
Read what they wrote28
Worked well10
  • The lecture notes were amazing. Generally the way today was structured was really good, I liked being able to revisit previous chapter and the explanations in depth
  • The notes were _really_ great. Again, I was too low energy to keep up with them at the intended pace, but for me this sort of material is a really effective way to learn.
  • Great reading
  • Exercises and readings were appropriate
  • The exercises!
  • I think Lois is by far the best lecturer we have had. Better than the others at getting people to participate; good at motivating how this is related to safety. I liked that he used the marker and paper to lecture rather than a power point filled with bullet points.
  • I never heard of this before, great. Great how this, SLT and Training dynamics essentially use the same notation and build on a sort of "toy models of DL theory" kind of concepts (a little bit less similar notation on training dynamics) but in general seeing this "trend" of work is super great.
  • I really like today's format. I think the pacing is also good. The selected readings were also relevant and useful. Sorry should have submitted this earlier in the day.
  • The lecture notes were the best resource and I really liked it as a pure mathematician. It helped me gain a good formal intuition about what the data attribution was and what the three suggested approaches looked like in math. Furthermore, starting the day with the husband killer AI discussion and brainstorming how to fix the problem led naturally to the concept of data attribution broadly (hmm something bad must be in the data cause where else would it get such crazy takes?). This informal inkling was then very nicely complimented by the formal maths that formalised the concept and suggested ways to compute the $\pi$ function from data thru parameters all the way to $\mathbb{R}$. Definitely enjoyed!
  • - having written notes was very useful - content-wise, the discussion about unrolling
Would improve11
  • Whiteboard was a bit hard to read, the connection to the causality shapley values could be made more clear
  • I didn't follow the discussion at the very beginning of the day. It might've been better to postpone that until after some of the reading/slides. Or alternatively, maybe I was just too tired.
  • this was great! I would just make more examples of how it has actually been implemented
  • readings first then explanations, since we can spend more time to understand the notation
  • The morning discussion was too dragged out. I think the exercises should've had more connective tissue with the ML.
  • I feel like it was good to have a small group of various backgrounds to advance on the problem.
  • I would like to know what happens after you attribute the data. There's a lot of evil on the internet so removing all of it before retraining seems difficult, and anyways it seems counterproductive for a model to not understand evil. I get the point for something like removing backdoors but I don't get other use cases. Maybe it would be more appropriate to make this 1/2 a day, since the data attribution subfield seems a little premature, and teaching 3 premature ways to do data attribution didn't feel more helpful than teaching 1.
  • There is too little vegan food (yesterday as well) so later in the day I become extremely hungry and it is hard to concentrate, it doesn't help to just eat the snacks in the kitchen either. The air quality in Cantor room is bad and also makes it hard to concentrate. I spend a lot of time to try to get a good intuition for what is going on, I barely got to the questions before it was time to reconvene. I might just have a low learning rate though.
  • During the morning we spent more than an hour and a half in a discussion in which people were guessing intuitions. Maybe people like this kind of stuff but I'd just preffer to have content. This time I think could for sure be used for following the lecture notes more thoroughly - again, those are quite good. A general trend in the intensive is that lecture notes tend to be quite good, we did like 30% of them during the day but I wish we did more. On the other hand, we spend time with hand waving discussions and sometimes classes (today the class was reasonable apart from the discussion in the morning, but in some other days classes took like 3h when should take 30-40min, like SLT and CompMech day). So yeah, I'd double down on lecture notes and I feel this would improve the understanding of everyone. Instead of doing a handful of exercises, give us the day to study the whole thing under supervision from professors there to help. One failure mode for the lecture notes is that if someone study them with the goal of "solving the exercise fast" (which happens here due to lack of time because time was used to something else), it's possible to solve the exercises mechanically because they're not super hard. If we had the whole day and were supposed to study ALL the notes - in a "I'm going to grind all these materials cover to cover 100% them", I think there would be enough time to actually stop some minutes to reflect about the meaning of the exercise and this would also be improved. I hope this was not a super confusing stream of thoughts, but yeah, I feel like iliad have the hardest part of a very good program figured out - the actual lecture notes - but then these extra activities kind of makes things worse
  • This was a good video when I learned about Shapley Value: https://www.youtube.com/watch?v=NBg7YirBTN8
  • I think sometimes the handwritten on the paper board / spoken parts of the day were a bit hard to follow because the main resource was a set of lecture notes. Sometimes these would be on the projectors and that made them impossible to read.
Changed my mind7
  • Not so much today specifically, but over the course of the last several days I've gained a greater appreciation for empirical alignment work, even if it's not what I personally am instinctually excited to work on.
  • more ideas for attributing data to behavior
  • No
  • I never knew this whole field existed so that was important information to get
  • This seems to be a lot more useful than anything else we have gone through.
  • Panel in the evening was super useful.
  • The extent of worrying phenomena (e.g. subliminal learning) that are only partially explained by theoretical results
Alignment in Practice Thu 20 Aug, Garrett Baker 25 responses · energy 3.7/5
Overall +0.16 as expected
Content +0.12 as expected
Clarity +0.80 better than expected
Motivation +0.96 better than expected
Exercises 0.00 as expected n = 3
Readings -0.04 as expected
Pace -0.40 too slow
Difficulty -0.68 too easy
Novelty -0.16 an even mix
Read what they wrote29
Worked well8
  • Readings were well chosen.
  • Having someone yap about a slice of what's going on in empirical AI safety was helpful.
  • already covered
  • I liked talking about what frontier labs are currently up to! I think this day should certainly be kept in because it framed things not terribly.
  • I liked the speakers, I think both had great personalities and passion for the topics they discussed. I appreciated that Aden's enthusiasm did not diminish during the afternoon slump. Possibly my pleasant feelings are because the concepts were easy to understand in English (vs. some complicated notation) and did not take too much cognitive effort to understand.
  • I wish the first lecture was a bit more as in it felt like just being read off of slides. Second half was awesome
  • Reading list and discussion was useful. I like the light brush on Borda and a touch on Arrow’s Impossibility Theorem. I wish we go into more details.
  • The readings & discussion about different approaches to AI control were useful and mostly new to me.
Would improve11
  • Last night at the evening session, someone made a really interesting point about how toy models should be motivated by experimentally observed phenomena in non-toy models. I wish today had focused more explicitly on those sort of connections between "alignment in practice" and theory, as opposed to just "alignment in practice" in isolation. Especially since control seems like it could have a theory component (e.g., one of the papers talked about how the strong AI was only allowed to send short messages to the weak AI, which seems kind of like interactive proofs from complexity theory).
  • Problem sets would be useful.
  • This should've been first or second day.
  • More perspectives would have been good. The fellow (Aden?)'s perspectives were good to hear but I assume they were more pessimistic than the averags knowledgable person's perspectives.
  • I guess today's session was envisioned as a bit more chill and not so technical from the start and it was a good interlude between all the maths heavy days. The one thing that worked very well was providing all the background lingo and knowledge AI safety researchers know well and use, and putting the whole development of LLMs in historical context. I particularly enjoying the way it was kicked off, with the presentation about what really goes on in RD at the labs, from the transformer architecture to fine tuning and RLHF, and the various alignment problems and approaches. The whole day was one of my favourites because it provided a broad conceptual picture of what actually is done in the wild.
  • I feel like this day is a bunch of concepts that people with epsilon familiarity with AI Safety know, but then yeah, if Iliad is getting people that never even did bluedot for example, it makes sense for the day. One way I believe this would be better is if this was in order like in the actual second day, or maybe cut this to half a day. I would've preferred to have one of the cut topics in place of this (like RL - it'd be mostly review for me but at least I review useful stuff - or AIXI - am not an expert and would benefit from learning more). In general, I feel like the order of "A1,A2,..." should be respected.
  • I think this day should happen possibly on the second day, as I feel like most of the concepts today have secretly been alluded to already/ assumed to be familiar to us and so it helped frame a lot of things from previous days. Today also felt more big picture, so I personally feel like it would have been better to get this larger overview before diving into the more concrete mathematical content e.g. SLT, MechInterp. It's definitely possible that I enjoyed today's material as much as I did because of the background I gained earlier in the week... Also, today's lecture felt more related to Launch day and it would have been better to have more time to stew on this.
  • I feel like there was some overlap with the first day. Also the llm training stuff was already pretty well known to me. Maybe some of the material can be merged.
  • would have it earlier in the program since it was more introductory
  • I’m quite torn because the class materials are important to know about, but the content is boring, as is the problem with the short-term alignment agenda. I taught a similar class before for BlueDot Impact, and I think you could rebuild the class structure and materials in a similar way because they had clearer takeaways about why the material was important. Maybe it is because this is mostly a theory course and I'm a theory audience (so I'm very biased), and most of the other classes are more exciting than this one. I actually think the first alignment class by Leon was pretty awesome and the way the material was structured were useful for a conceptual day, and you could perhaps follow a similar structure. Even with the SLT class, which I didn’t fully follow because it was too fast for me, I still felt like it was more worthwhile than today’s class materials. I think the readings for AI control were useful and well selected. I actually think that the World Models, Computational Mechanics, and Data Attribution days have been among the best-taught classes so far, both in terms of content and format. However, I’m also familiar with the underlying concepts by proxy because I come from a control theory background. They were very clear about what to do while giving you the freedom to learn in whatever way suited you best.
  • Might be a good idea to move the AI alignment in practice day earlier on? (At this point, much of the content was mentioned or at least hinted at during the other tracks.)
Changed my mind7
  • I was indifferent to the control literature, but after reading some papers on control I feel I believe control is much more important and worth spending time on.
  • No
  • AI Safety is hopeless. I am looking into becoming a monk, and if that fails I will become an electrician.
  • I realised I should've stopped betting against AI a long time ago and sort of recalibrated my epistemics.
  • I remembered that empirical work is cool
  • It made it more clear why safety work is important, how the models are so bad in some respects and the role of the big AI companies in contributing to the lack of safety
  • I'm wondering if theory is completely not useful at all in practice and that's why people now seem to care about AI for Societal Impact type of agenda.
Other thoughts3
  • The preliminaries on transformer were super nice! I kinda think this day should be taught earlier in the program
  • So _this_ is what empirical alignment research means. I retract any previous comments of mine to the effect that Iliad is more empirical than I had expected.
  • Sorry for only submitting this now...
Launch Day Mon 24 Aug, Leon Lang and David Udell 21 responses · energy 3.9/5
Overall +0.38 better than expected
Content +0.33 better than expected
Clarity +0.10 as expected
Motivation +0.43 better than expected
Readings 0.00 as expected n = 8
Other activities +0.67 better than expected n = 18
Pace +0.10 about right
Difficulty +0.05 about right
Novelty +0.62 mostly new
Read what they wrote15
Worked well1
  • Both the quick overview of programs & the presentations were very helpful (and moving the presentations earlier in the week was a very good idea, there are several people I still want to chat with about their ideas)
Would improve11
  • Just make launch day better, or give up on it. 4min was too few, I don't even know how it is possible to form some impression about people in 4min presentation aside from metrics like "good at public speaking". One idea would be just include the whole day to presentations and then more time for everyone. Or make people record presentations of at most 10min in a website and send to you guys, so you guys can see (if the idea is to get points for the fellowship). In general this day was somewhat communicated poorly, I don't know. Content day would've been way better. The morning AIS overview of funding is perhaps helpful for newcomers but I bet most people here were familiar with eveything there (and if not, it's on them). For Ryan Greenblatt's text, it's just too shallow, if it is to do something with the content of the text, just give us more time and we read AI2040 and discuss properly (although I also think a theoretical day would be better).
  • I don’t think I learned much from the presentations, I would consider dropping it.
  • 4 minutes was harsh for talks.
  • The whole presentation idea was a very nice breath of fresh air from the usual technical content we go through. It sort of put taking part in the intensive in perspective and made us actually think about immediate future steps like projects grant applications etc.
  • I would move the presentations for the last day of the Intensive.
  • I think the primary thing was that I felt that the presentations could've been organized better. I think initially on Discord we were told that we'd have 10 min to give a talk, but this was cut to 4 min day of because more people than anticipated wanted to talk. Perhaps if there was a poll or something on Discord then this could've been better handled.
  • Not sure if this is a feedback for the morning session, afternoon session or today as a whole. David's session was pretty useful for people who are newer to the field which might be most of the participants. Morning session with Leon was fine, I would include the AI Control reading material from BlueDot Impact course since it describes the agenda in a way that would capture broader audiences. As for the afternoon, there were a lot of interesting ideas and overall they are useful talks. I appreciate that Zach paid attentions to the participants and ask a lot of good questions. I also wish other teachers were doing something similar.
  • The reading was near useless - too scant on details (but it is also bad to be too long). Today was very uncoordinated - most students didn't return on time after the break (no gong was hit) delaying things a lot. The discussion was weak (small group discussions would've been better). The launch day itself was disorganized and the main lead on organizing seemed way too eager to just cut people off from presenting. We should've had more time each.
  • I think the problem today was that because I got much less time than expected (I expected 5-10 minutes) and everything felt rushed, I was taken a bit aback and the quality of my presentation suffered tremendously (like, I felt I didn't have time to put the presentation on my screen as well as the big screen, which meant that I lost my anchor, and I felt I had to rush, and then it felt like almost no meaningful words came out of my mouth). To improve this: Next time, figure out a few days before who wants to present, assign slots, and ideally, have all the presentations on one computer (as in, have people send them to you the day before).
  • It would be nice to schedule some time to work on the presentations. Now I feel like they were quite rushed.
  • The ordering system
Changed my mind3
  • Over the weekend I decided to commit to AI safety.
  • There are many directions to proceed
  • Learned & thought about some research directions (some feasible generaltions of DLNs and what is known about their training dynamics etc.), thought more about career planning
Agent Foundations Tue 25 Aug, Aden Power 29 responses · energy 3.5/5
Overall +0.62 better than expected
Content +0.55 better than expected
Clarity +0.72 better than expected
Motivation +0.34 better than expected
Exercises +0.33 better than expected n = 24
Readings +0.56 better than expected n = 27
Other activities +0.45 better than expected n = 11
Pace -0.14 about right
Difficulty -0.14 about right
Novelty +1.03 mostly new
Read what they wrote32
Worked well8
  • The game theoretic part was super interesting, thank you for allowing so much time for discussions during the lecture! Also the game was super fun!
  • Great teacher
  • Aden is an incredible teacher, I really enjoyed today!
  • Aden put lots of time into making slides, materials, lists or readings and exercises for every topic!
  • The game at the end was really fun, it was cool to see an actual game with an evolving optimal strategy in practice
  • Liked the readings with multiple choices.
  • The game was fun and engaging.
  • The logical induction mini track!
Would improve13
  • Just do more days on Agent Foundations. It is too much content for a single day. In general, it was nice and less hand wavier than other days. I feel like with a single day it's hard to do better. Again my general recommendation of "just let us do lecture notes" during the whole day and be available to help applies (at least to me)
  • Would've wanted more discussion about actual applications of this to safety.
  • People spent a lot of time being confused on definitions, maybe keep important definitions on all relevant slides.
  • Great teaching! Just confused about the agent foundation field, it all seems very abstract and it is unclear what the point is to me
  • Aden is such a good teacher that I'd rather have him teach than me read or do exercises.
  • would be interesting to know some main result in lecture from each of the subareas. still not very sure if I should spend time in this area.
  • More concrete examples, especially for embedded agent. I’m still trying to think of a concrete example of where dual agent theory can’t explain something that embedded agent theory can explain. A concrete example of the game we played at the end before it started.
  • group exercises
  • More structured overview of subfields of Agent Foundations, since it is not really a single field like the other days. (If at all it is must be a single day, participants should have a way to go much more into depth into one approach comparable to other days, rather than just a shallow overview of many topics. There should be mentors knowledgeable about each specific subfield so the participant can learn from a person (which is much of the point of ILIAD Intensive over just reading articles and papers online), and which mentors are knowledgeable about what should be broadcasted, loudly.)
  • setup was really good! lectures were quite engaging and I think agent foundations as a field probably benefits from not forcing depth of formalism in a survey where it might struggle to improve understanding
  • I think I would rather prefer a Reinforcement learning day to be honest.
  • Already gave feedback individually. Might be worthwhile to checkout Michael Dennis' PhD Thesis for the self-referential claim. Slides 14 and 15 are quite helpful and some of the best part of the day. Seems like I'm super biased towards today materials as I'm actively working on it. Aden is a pretty amazing teacher. I also really like the approach of learn whichever way that's best for you.
  • It would have been helpful to cover more mini tracks within agent foundations in more technical detail (e.g. AiXi, natural latents etc.), but I assume this would take more than one day?
Changed my mind8
  • Agent foundations seems boring and very formal
  • not sure about agent foundations, will have to look into each subarea still
  • I feel agent foundations might be a new interest for me and I am glad I was introduced to it :)
  • Agent foundations is better than I thought.
  • it is worthwhile to create formalisms for analyzing possible scenarios for ai safety
  • I’m gonna pay more attention to open source game theory now
  • I liked Löbs theorem!
  • I'm starting to wonder that alignment and safety is not something that we can fully solved and we have to keep working on it even we are in a short timeline take off scenario.
Other thoughts3
  • The game was very fun.
  • N/A
  • Seamus and Pablo raised a really good point that agent foundation work in the opposite way of typical scientific discovery. Typically in Physics, you solve some specialized thing first then generalized. So I'm uncertain about the agenda. Nonetheless, I don't think I can find myself working on anything else but agent foundation since I think it's the work that is most intellectually satisfying.
Debate Wed 26 Aug, Stephan Wäldchen 25 responses · energy 3.4/5
Overall +0.80 better than expected
Content +0.80 better than expected
Clarity +0.48 better than expected
Motivation +0.80 better than expected
Exercises +0.15 as expected n = 20
Readings +0.44 better than expected
Other activities +0.29 better than expected n = 7
Pace +0.16 about right
Difficulty +0.36 too hard
Novelty +1.44 mostly new
Read what they wrote27
Worked well8
  • Gong usage was pretty good although I was late to one because I don't think it was hit (maybe I didn't hear it?).
  • group readings and exercises
  • Rigorous schedule
  • The exercises were great and not hard to follow and the structure of today was quite amazing. Stephan also communicate well and bery structure. Honestly one of the best teaching day so far!
  • Great instruction, seemed very knowledgeable and enthusiastic
  • I really liked that Stephan actually made good use of the day (few breaks, lots of material)
  • I think the method of having people go up and present the paper is great way to communicate the key ideas from multiple papers without having everyone read all of them.
  • The exercises were helpful and the content was great :D (and it was nice to have a whiteboard lecture)
Would improve11
  • I think Stephan should've went slower on some things (some of my friends were pretty lost). Also, maybe making the big picture of debate clearer would help. Some of the exercises were kind of trivial. I didn't learn much from the readings.
  • I found it quite hard to understand how the toy models of debate and protocols are scaled up to frontier model levels, so a bit more words on how this approach is expected to play out for alignment would've been good.
  • This was one of the days I was most excited about. In the end we had like 2 sessions of 15min to read the lecture notes (which are the things that are actually detailed with theory of debate), and I guess the teacher expected us to get all the details in these 15min? I just couldn't. The other papers were understandable and the classes were fine, but I just don't think they would do a better job than good lecture notes. I guess the main suggestion is improve these lecture notes so they are a little bit more self contained (for ex., I didn't know all the definitions of complexity spaces used in the notes, and the notes just gave some handwaviness definition of them, but if you're actually proving stuff about something the object should be clear).
  • spend more time on group exercises
  • Today was amazing! I'd definitely change the morning readings (AI researcher). I felt the 3 papers are in the lower quality end and they didn't really fit into the "trifecta" framework
  • This is hard to say because Stephan is a wonderful instructor and the course structure is laid out really well. I don't know if I'm biased since I just realized that Debate is using a lot of methods from game theory and computational theory of mind which I'm quite familiar with. The exercises were great and not hard to follow and the structure of today was quite amazing. Honestly one of the best teaching day so far!
  • there were some things squeezed in such as the final aisi slides, didn't follow them entirely. also 10 mins for the exercises isn't enough
  • I think I'd have preferred getting a better idea of what *work* in the field actually looks like, and see a bit more the connection between theory & practice. I think the discussions after the readings could have been improved - hearing others discuss papers that I haven't read isn't super interesting, especially because no one really wanted to. I think from a pedagogical POV, Stephan should just summarize the papers.
  • I think today was a great topic and I really enjoyed it, but I think many people were (understandably) distracted by trying to fill out the interview for Garrett. Maybe I would try to release it as a timed test or something in the future...? Maybe with people having the questions in advance a little to reduce stress (sorry I am worried my peers will hate me for suggesting this haha so I am trying to balance my suggestion in favour of what they might want me to say for their benefit)
  • I would have skipped the first reading session: At least the math Lean paper seemed to be summarizable in five minutes, so imho the time could have been used more efficiently for other things, but I understand that this might just be my preference
  • Very nit-picky, but I wasn't a big fan of the VeriPlan paper (but it was still useful to set the ground for debate)
Changed my mind6
  • Scalable oversight seems very promising
  • Debate has a super clear and strong theory of change
  • I now understand why people in Game Theory and MARL decided to work on this topic.
  • improved liking of debate
  • Today's safety agenda seemed the most promising out of all the things I have seen so far
  • Without having any knowledge at all debate just sounded weird to me. Seeing exponential complexity to polynomial was a great motivation. Its actually really interesting ! One of my favourite topics
Other thoughts2
  • A lot of self-play method has been used quite significantly in debate and I'd like to investigating debate a little more!
  • Interesting mix of activities today. Generally, I think it would be helpful if the days carried more descriptions in the calendar, eg what the prerequisites are expected, what topics will the day build on. That way we could do some preliminary reading or something. This topic was definitely super difficult to anyone who hasn't done any computation/complexity theory before.
Steganography and Backdoors Thu 27 Aug, Stephan Wäldchen 24 responses · energy 3.5/5
Overall +0.58 better than expected
Content +0.50 better than expected
Clarity +0.42 better than expected
Motivation +0.79 better than expected
Exercises -0.05 as expected n = 21
Readings +0.26 better than expected n = 23
Other activities +0.44 better than expected n = 9
Pace +0.25 too fast
Difficulty +0.17 about right
Novelty +1.08 mostly new
Read what they wrote17
Worked well7
  • Stephan is a great teacher!
  • Both the choice of content & the two talks!
  • group exercises worked
  • group exercise, interesting readings
  • The game was so cool! The topics presented today were interesting and I will look into them more.
  • The lecture was amazing: Fun game as an introduction to the topic, super clear and structured explanations. Very well chosen content, level and depth.
  • I really like the table schedule for the whole day!
Would improve7
  • Again, do good lecture notes and people study them in the day in intensive, with supervision, and that's it. Alternatively if this is too extreme, just don't make the day feel like a bluedot course meeting, like funny game then not a lot of time to actual reading on my own than empirical paper reading for 5min and discussing one idea that honestly we just take at face value because a paper is not readable in 5min, etc. This day was less confusing than yesterday but still kind of terrible, the ideas are for sure important and I even got more security/formal verification/complexity theory pilled with all of these, but the implementation of the day is not great. Louis talk was amazing, not because the talk was particularly interesting but this idea of slowly building up ideas to solve huge problems and figuring out which problems are even worth trying to solve beforehand is great. I generally think his mental model about stuff is correct and agree
  • Maybe more exercises (even take home) that cover the content from the first part of the day?
  • not enough time for reading papers, doing exercises
  • Great material and variation, just a bit too fast paced
  • I feel like today we were covering too much ground and I was getting lost very often. The day felt very dense.
  • The lecture was amazing, but the talk was super confusing to me: Please provide a big picture overview first and then either go into detail or leave topics out. Leaving lots of open ends while at the same time not mentioning which topics are central makes it impossible to follow. The definition of f(M') was too vague (missing domain and codomain) and could only be reverse engineered via its relation to C(M').
  • Exercises: I feel like the exercise text can be improved, I felt like the "relevance" was a bit hard to see. Also solution d) is missing (not like there's a lot one needs there) Readings: Not enough time. I think it would be useful if Stefan was a bit more specific on what we should read given the time constraints
Changed my mind3
  • I'm more ARC theory research pilled (their new stuff)
  • feasibility of planting backdoors in LLMs
  • understand how steganography is applied better
Physics of Deep Learning Fri 28 Aug, Charles Renshaw-Whitman 16 responses · energy 3.3/5
Overall +0.25 better than expected
Content +0.50 better than expected
Clarity 0.00 as expected
Motivation -0.31 below expected
Exercises -0.06 as expected
Pace -0.06 about right
Difficulty +0.06 about right
Novelty +0.94 mostly new
Read what they wrote12
Worked well2
  • The freeform study. High level intuitions. And how to think about hard things in an easier way. I particularly appreciate the first lecture on how to think about kernel trick.
  • I found the part the on NNs that learn features rather than using fixed ones the most useful.
Would improve6
  • I'm really biased but I really like Charles'style of teaching. The notebooks can be improved and have more exercises.
  • No more slop exercises. They are demotivating. Quicker start to the day, I think we spent too long talking about the plan and outlining.
  • I feel like Charles largely "vibed" this and didn't have a clear pedagogical plan here. I felt like this could have been a lot more structured. The note books are pretty solid, but I think notation could be simplified a lot
  • I didn't understand why I should care about feature function, so maybe that should be emphasised more. Generally I like to learn something because I need it to solve a problem; today I feel like the vibe was that there is a mountain of mathematical techniques and ideas I need to learn to get to the forefront of the physics of deep learning, and I should trust you guys to lay out the path for me. And I do trust you guys, but there might be a more directly motivating way to introduce feature vectors. I think Louis's presentation on data attribution was especially good at motivating why it is important.
  • Charles is a great teacher and the notebooks are great (though being honest, I fully did the first one and skimmed through Notebook 2 and 3). I think if those notebooks become (more) self contained lecture notes, this would be one of the best days.
  • I would make the lectures more mathematically rigorous and prepare slides walking the participants through the material. The lectures we were given were pretty unintelligible to me and whenever a mathematical question was asked it was just brushed off.
Changed my mind3
  • I had a conversation with Daniel Chiang and we disagree on studying rational vs irrational agents and his argument convinced me why we should care more about studying rational agents in mechanism design.
  • I now think that this is extremely dual use and I don't really see how this connects to safety.
  • Didn't know why people say NTK is useless, I get the general vibes now
Other thoughts1
  • It's unfortunate that this is the last class. I'm gonna miss the classes and I'll rework the exercises on the days I got sick.

Reading the bars

Every bar is the answers themselves, laid on one axis with the middle of the scale at the centre line. Left of centre is worse, right of centre better, and the widths are comparable from row to row.

much below below as expected above much above

Pace and difficulty are a direction, not a quality: the centre is the target and both ends are a drift away from it.

on target one step off two steps off

A row says n only where fewer people answered it than answered the day, such as a question about readings on a day that set none.