Alec Boyd Not you?

By cohort

April 2026

November 2026October 2026September 2026August 2026June 2026April 2026

Rated 1 to 5 in the absolute. Not on an axis with the signed means of later cohorts.

AI Alignment: an introduction Sat 4 Apr, Leon Lang 14 responses · energy 2.9/5
Content 3.9 of 5
Teaching and format 3.7 of 5
Difficulty match 2.6 of 5
Energy 2.9 of 5
Read what they wrote35
Worked well12
  • The amount of time per reading was good, and the reading guide in the doc was helpful (as were some of Leon's explanations)
  • Discussions
  • I liked that we took the time to do many 1-1s (ideally with everyone, but that unfortunately didn't work) to get to know the other participants.
  • Guided discussion questions! and freedom of interpretation
  • The idea of 1-1 and breakout discussion groups.
  • The individual reading then discussing it amongst ourselves was a good way to learn and learn from what others had understood from the same articles/papers.
  • Meeting other participants, having discussions about perspectives on AI alignment.
  • Meeting people in the morning was really good, eventually had issues with not having meetings synchronised, so slightly more structuring might help. Discussions on the reading was interesting
  • Reading time was nice
  • Discussions, the people
  • 1:1s were fantastic!
  • I liked the content! I also liked the discussion in small groups.
What they took away11
  • I think the general background was useful, thought I wouldn't attribute this to one specific resource
  • I don’t know the term CEV somehow
  • Reinforced concepts of alignment as a problem and definitions
  • Alignment target: there's inherent problems to how we can formulate what we want. Intent alignment: how we can tell Agent to do what we want it to do. Not what the right thing to tell the agent is. But what is it that we want it to do? CEV, alignment targets; alignment problem decomposition: specification gaming. Shard's theory.
  • The view that AI is acting in a context and that we are actors of this context. Such as when AI agents wait for our prompt before acting.
  • I learned many new concepts (and names of ideas) in alignment - eg inductive bias. Helps to shape arguments about these things and identify problems to attack.
  • Refreshing if basic ideas was good for me
  • There was one article I had not read yet, the one on tool AI vs agentic AI. That was good to read
  • Maybe reading about process oriented learning or training stories
  • Singular Learning Theory, also met a person whose interests me greatly.
  • The paper on how agent AI is more likely, even though I have criticisms of it but this led to interesting discussions.
Would improve12
  • Not all of the readings had useful information to length ratios (e.g. the inductive bias and rocket alignment posts), but this could be got around with Claude summaries
  • Grouping people / systems that help everyone contribute in discussions
  • The 1-1s were a bit chaotic and we didn't have time to do them with every other participant because of that.
  • More structure in information presentation, more organization, slightly lower quantity or literature or higher density of information/simplicity, more context surrounding literature and non-academic sources/critical evaluation
  • 1. Have a more structured way to allocate participants for group discussion. and yes, having moderator involved in the group works nice! 2. The food today was too spicy. 3. Reduce the number of discussion times? 4 times of reading and group discussion back to back is tiring. And also having consistent discussion group set is better than keep swapping it since we can reference points said earlier and synthesize insights better.
  • I would give more time to read the articles and also have a whole group discussion after the smaller group discussions.
  • A lecture summarizing some of the concepts might have been helpful, in addition to individual reading and small group discussion.
  • Questions on the reading maybe could actually be closer to just suggesting to discuss your thoughts from them, as to me 40 minutes for the readings was not much, so just thinking about them is probably the best option. Increasing the 40 minutes might help, although clearly there’s not much extra capacity in the day schedule.
  • It feels like it was a bit too basic/too much I have already seen before. Not much novel discussion. But good to get people up to speed. I would have preferred more reading/personal learning time and short general discussions with the full group than these longer discussions with split up groups. For getting to know eachother it would have been nicer to have a round around the table quickly and then just dive deeper immediately. Also maybe just a brief overview/presentation of the material might help the quality of the discussions.
  • I have ADHD so my ability to read for a long time is very diminished when compared to others. I'd prefer video based content, ideally video recordings of presentations, since I usually need to 2-3x them in speed to be able to focus on them and not be bored by the speed (which is why I have a very hard time listening to real world talks as well).
  • Seating was a bit uncomfortable. Lunch was too spicy for me (I'm sorry 😭)
  • Ideally having the instructor assign somewhat different groups for each discussion moment would have been better so we don’t end up discussing with the same people each time.
Intro to ML Engineering Sun 5 Apr, Julian Schulz 12 responses · energy 3.3/5
Content 4.1 of 5
Teaching and format 3.7 of 5
Difficulty match 3.0 of 5
Energy 3.3 of 5
Read what they wrote34
Worked well11
  • Just learned a lot, tinkering yourself with the notebooks was very useful
  • I liked the applied section (track B) and then the lecture after with tutorial.
  • Self-selection into tracks makes a lot of sense for this topic, given the diversity of backgrounds.
  • The track B tools we learned about and implemented will be really useful I think.
  • Closely scaffolded coding tutorials
  • Format and teachers were good. Track B content was also useful to see, as well as some of the RL content at the end
  • Breaking up into multiple tracks depending on your level was good. I was in Track B. A mix of practical-work-through-these-exercises and then having a lecture in the afternoon was good for energy management.
  • Julian’s energy was infectious. The community did not seem competitive or shame-driven.
  • Letting people in Track B work on the instruction individually > pair programming
  • For Track A (Julian): great combination of lectures and notebook exercises.
  • Exercises were good, pair programming was good
What they took away12
  • So much! Just generally going over all these structures, from the bottom up, was extemely useful
  • Lecture part on reasoning models was really useful and also the tutorial of implementing an attention layer.
  • Transformer architecture
  • How to work with Runpod probably, and a quick overview of RL
  • Probably runpod
  • familiarity with the ease of vast ai
  • I liked the very end about RLVR
  • How LLMs' chain of thought metholody are trained on verifiable outputs such as maths and coding.
  • I got lots of links: I think/hope more learning will happen later, when I have more unrushed time to go through them all, than actually happened on the day.
  • remote SSH configuration with vast.ai; supervised finetuning and RL.
  • How RLHF works, and how to train reasoning models - at least the beginning of this.
  • Reminder of some practical details and implementation details for transformers
Would improve9
  • Smear this over yesterday and today!
  • Understandably the acquisition of a GPU cost a lot of time but that would be great to fix. I would have had enough time to finish track B’s content otherwise, which would have been ideal.
  • Better setup and exercises for track B. Lunch was too spicy, I'm sorry 😭
  • The logistics of the runpod credits
  • I would have liked to see a coding session, especially if it's an experienced software engineer showing how they use an agent to get to a solution and work through a problem. In Track A there was a lot of conversation about agents at the end and how useful they are even for software engineers (per the TA of that track).
  • I would have liked to get the document with the links describing the content of the day, before the morning of.
  • 1. More assisted individual work: I found having more TA accessible rather than just relying on 1 instructor is useful. Less pair programming: pair programming makes it more high-cortisol and challenging. 2. The pace of afternoon session - Lecture 2: Julian spoke too fast, rush too much, I would also swap slides with interactive website (Claude can make one) to explain the structure of transformer/attention rather than static images/slides. He links to a LLM visualization website, which is a good source; but doesn't replace having an interactive website to visualize the architecture itself > slides. During the lecture he also asked people to raise their hand if people know/don't know a material: this honestly wouldn't be accurate since social pressures would drive people who don't know a concept to not raise their hand. It's much, much better if these lectures can be released 2 days prior: how it works can be: you make a form that asks key questions over the day's materials: from 0 to 5 how confident are you with Concept A, B, C => At the end of the survey you share the link to the day's content document. This would make more people to fill out the survey and more attentive to their own weaknesses, and fill out any knowledge hole before the lecture; furthermore, it's better for people to self-learn/review concepts and come with better questions/better concentration. 3. The 1.B was good, but configuring remote SSH took an entire morning and was very inefficient. Could this have been prevented?
  • Hope to learn more about RLHF and get some hands-on experience?
  • More time on transformer pair programming, slower presentation of details around LLM training (as I found the content on RLHF and RL largely incomprehensible where the setup details were not so clear from the slides - fully written notes would have been a lot clearer to me)
Other thoughts2
  • Would love to be able to access content earlier, so I can prep the day before. And also the days I cannot be there, would love to be able to still ask questions in discord/review the work early.
  • We should check out more Vietnamese or Chinese restaurant please.
Mechanistic Interpretability Tue 7 Apr, Julian Schulz 13 responses · energy 3.3/5
Content 4.5 of 5
Teaching and format 4.1 of 5
Difficulty match 3.4 of 5
Energy 3.3 of 5
Read what they wrote33
Worked well13
  • Pair programming, much of the teaching content and demos (generally really enjoyed it!) The regular breaks and exercises are very important because it seems really hard to keep absorbing new ideas after an hour or so
  • 1. 10 min break! 2. Time for independent experiment/exploration/reading then re-group for short discussion.
  • We got a good overview of MechInterp techniques.
  • Lecturer so good
  • I think Julian just did a really good job
  • The lecture content and resources was really valuable.
  • Lots of breaks, discussion time, mix of exercises and lectures (time to think about materials/slides in lectures)
  • I liked the format, which included programming, discussion, and lectures
  • The presenter was great and their content was engaging. The online tools to play around to find out what's happening inside LLMs was a great way to better understand the mechanisms. The resources provided, even the papers for a conversation at the end were interesting.
  • The mix of theory and exercises was nice.
  • The lectures were great.
  • Julian is engaging and capable and impressive
  • the interactive questions with tier 1 and tier 2 worked well!
What they took away11
  • Clearer views of SAE methods and circuits
  • 1. neuronpedia 2. sparse autoencoders breakdown
  • Circuit Traces (and techniques for finding circuits in general) + the (unfortunately too brief) discussion about arguments "against" MechInterp
  • Very low level structure of mech interp techniques
  • Saw the technical details of mechinterp approaches
  • ACDC
  • SAE implementations and variations
  • Playing around with the SAEs and circuit tracers was informative (e.g. to directly see how messy it is)
  • How LLMs organise related information "close to each other".
  • How SAE’s are trained and how circuits are detected
  • I collected a lot of links to read later! I felt I got not only a clear snapshot of the current state of play, but also the recent history of how we got there. I feel I gained something like dialectical fluency, i.e. I now know the meaning of lots of proper nouns I might hear tossed around when chatting with other researchers, whether I like ever use them myself or not.
Would improve8
  • 1. having more time for coding problems! although this would not apply for the days going forward. 2. maybe have a buddy system: pair 2 people with each other to discuss the materials learned up to now (~after 3-4 hours?) => they can resync and reflect on the knowledge better, ask and answer trivial questions, and deepen their understanding to ask smarter questions to the instructor!
  • More time for the discussion about the usefulness/ToI of MechInterp
  • It was quite good, I would just have loved to have more time to prep, I could not follow most of it
  • There wasn’t much time for the discussion or exercises. I think I would prefer to just focus on the lecture content and have breaks in between rather than squeezing in exercises.
  • more time for discussion ~15-20 min is good for a few topics, shorter is fine if the content to discuss is more limited
  • The classroom layout sometimes makes it hard to see the board. Otherwise I would like to see some live coding of the exercises (the notebooks), so that we can better connect the theory to practice.
  • Maybe more time for coding exercises
  • I found a day-long whistle-stop tour uncomfortable. By the afternoon, I was fogged out and not taking in new information. I felt like I was trying to look polite while waiting for it to be over, so I could read a few of the many papers that flashed by and follow my own train of thought about the ideas in the them. I want to make clear that I have a lot of respect for Julian and valued his efforts! I love that he made a doc for us with the slides and papers and exercises in it, which will make it so easy to dive back in later.
Other thoughts1
  • I think generally discussion in the sessions should be more limited and mainly just focused on asking clarifications of what is said - I felt probably too much time today was spent on discussions better done more privately
Deep Learning Theory 1 Wed 8 Apr, Zach Furman 12 responses · energy 3.1/5
Content 3.6 of 5
Teaching and format 3.0 of 5
Difficulty match 3.3 of 5
Energy 3.1 of 5
Read what they wrote33
Worked well10
  • Zach was very calm and clear and humble. His pace was slower than Julian’s, which I liked.
  • Just a nice overview, more depth would have been ok for me/more targeted exercises or where to find them. And a bit more background on why this is important/historical background of the subjects would be good!
  • It was fun to have some math exercises to do, as opposed to just reading and listening.
  • Readings
  • Exercises were useful and main presentation was good, zach is entertaining to listen to and answered questions well
  • The lecture at the beginning setting everything up and the discussions at the end were useful for bug picture context, though I think some of the detailed Solomon off readings were a bit too dense to take away much from (the exercises were I think, in the other hand, a bit too easy to be super informative as well).
  • Some of the more mathematical readings were a good review of central learning theory concepts.
  • Good presentation and interesting readings, also nice to get hands-on with some exercises
  • the instructor is very kind and helpful!
  • The content is interesting and seems to be generally good background material. Doing the maths exercises was nice. The talk this morning was quite nice, but probably more detail on what is deliberately imprecise would be good (perhaps a note to say that a slide is not going to fully make sense, but paves the way for later).
What they took away10
  • Probably the nicely phrased final reading, 8.4 of the Cambridge textbook, on learning and cryptography as opposite processes
  • How these concepts tie into AI safety
  • The idea behind Solomonoff induction, and its relation with simplicity bias. Also the way in which NFL theorems can razor-away certain justifications for the performance of learning systems.
  • Solmonoff theorem proof
  • solomonoff induction schematics
  • It was my first interaction with Solomonoff induction - it was useful to understand what was going on, and especially to get results on universal behavior of various priors.
  • Solomonoff Induction and prior
  • approximation, generalization, optimization. the instructor's explanation of Solomonoff induction and its architecture.
  • Solomonoff induction theorem and some further intuition for what the various background assumptions are (although I need to figure this out still more for myself - I’m at least aware of having a blind spot here)
  • Solomonoff Induction and universal inference
Would improve12
  • Across all the days (not Zach’s fault!), I am finding the chronic enforced togetherness trying, by which I mean both the large number of bodies crowded into the room and the effort to follow the same train of thought at the same pace as a large group of more vocal people. I crave more independent time and space.
  • Probably the content a bit more structured, more specific exercises, less reading on our own
  • Maybe reading/discussions of the reading material could've been done in smaller groups again, like on the first day. Although perhaps the material is less well-suited for this due to being more technical.
  • More assertiveness over people is definitely needed, Julian was a good model for this who would more strongly elicit people to say where they were up to and stuff
  • Either just give intuitions, or make the maths very rigorous. The middle path was not great, the exercises were quite confusing due to the mixed and confusing terminology, and the lack of prior mathematical rigor.
  • more defined pedagogical goals going into the sessions, and scaffolding for this content. restricting to either conceptually overview of key results or narrowing the scope would be useful. breaking readings into chunks was not very effective when total understanding in a limited time is not possible and the main goals were not clear.
  • Maybe this will come later, but I think it would have been useful background context to see up front what people currently doing learning theory in AI are up to and whether/how this is useful
  • Some of the material might have been presented in a lecture, highlighting the most important findings - but ideally with a different lecturer. For my personal tastes, Zach was a bit too disorganized and confusing with explanations of what should be precise concepts.
  • Maybe have slightly more structured discussions of the readings
  • 1. Randomly assigned pair of 1-1 Q&A for 10 minutes after every 1.5 or 3 hours of content or a breakout group of 4-5 people and a TA to facilitate any Q&A. We desparately need, need this please. Reasons: - if we have 1 big group and 1 instructor, if one person asks questions everyone else queue. - Also, some are just not comfortable asking questions when there are many people. that's why you can notice there's almost a consistent group of people who keep asking questions and the rest who don't speak much. I'd want to hear more perspectives/questions from the rest of the people too, and want more psychologically encouraged to engage and ask more questions. I'd further argue that if things keep going this way, there'll be certain tension and conglomeration where people more active in class will form clusters, and the rest form cluster. Not good for class synergy. 2. Release the materials 1 day earlier, please. - For example, I'd expect to have materials for Day N + 2 on Day N. - Why: the materials are dense, we barely had any pre-requisite reading focusing on each of these concepts. Meaning, unless someone has stumbled upon this before, it's their first time reading/learning the concept. Since we have quite a big variety of backgrounds, I find it highly infeasible this occurs for the majority of people. Therefore, the least you can do is to give people the slides content, or more simple sufficient, having the instructor gives out bullet points of concepts covered that day. We don't need exact reading list or slides if that's helpful. We want to gauge the main concepts, so we can prepare better and come armed with more context, and engage more meaningfully with the materials. This also flatten out the bias: making it more fair for all class participants and neutralize the effect of having seen the materials before leading to better performance in class. Otherwise, the graduates offered to the post-residency fellowship would skew to almost all people who have already had prior exposure to the content, or pure genius who can picks up on learning immediately on the spot. Do you think this would be unfair for people who have a different background, different learning method, and different knowledge base about these, but who are equivalently ambitious, hard-working, and can perform better if pre-context and independent preparation for the materials ahead of class? the point of the class should be that students have the chance to meaningfully ask sharper questions and have it resolved, or meaningfully discuss wth fellows. If I'm just learning everything for the first time at class then I'd just prepare to do it on my own at home, I think I'd learn better that way. If you feel meh and negligent about the first two points I shared above, and would prefer letting the class be as it is -- why are you spending all this money funding for us to come here and learn if not taking into feedbacks and try experimenting? after all, we still have 3 weeks and the materials are only gonna get harder. 3. The room is too hot
  • I wasn’t a huge fan of the reading for Legg’s thesis - I think it wasn’t precise enough on a few points I’d care about a lot (e.g. what was a probability distribution or not, exactly what was included or not), so it lost time getting intuition for this where trying to figure out the definitions was not clear. Perhaps a handout sheet to state just the definitions plainly would be helpful, particularly for then using these for the exercises
  • The presentation of the material could have been better, the presenter is clearly extremely knowledgeable but didn’t seem to be so on top of this particular material.
Other thoughts1
  • Probably still less conversation in the class about questions or comments not directly clarifying the content would be preferable. I think questions of where people are at in reading maybe should be done more explicitly to fully check on positions (where this is certainly my fault to an extent for not answering)
Deep Learning Theory 2 Fri 10 Apr, Zach Furman 11 responses · energy 3.5/5
Content 3.9 of 5
Teaching and format 3.5 of 5
Difficulty match 2.9 of 5
Energy 3.5 of 5
Read what they wrote29
Worked well10
  • I found Zach's blogpost (and the content in general) very good
  • I liked the introduction and some of the readings
  • presentation
  • The food was good, and the office space was nice and well-suited for the starting lecture.
  • Started out feeling slightly chaotic, but all the concepts fit together by the end, through the readings and group discussions.
  • Hard cutoffs of readings
  • I found the content very interesting and useful
  • First reading from Zach was excellent, and would have been good to have read yesterday (in general, more time with readings from today would have been preferable to some of the readings/exercises from yesterday).
  • The readings were fun and the tables with 3-4 people works better for discussion.
  • 1. The reading was nicely curated and built our intuition much better. The order for the materials of yesterday and today should have been reversed. 2. Having TA in each group really helped. Overall much more insightful than yesterday for sure!
What they took away11
  • The deep-learning-as-program-synthesis hypothesis and representation similarity/convergence
  • That there are different hypotheses for what pressure creates generalisations
  • range of approaches and results to study the ways models represent concepts.
  • Getting a sense of why we might suspect that DNNs implement programs, and what that might mean for how to study them.
  • Perspective of softly enforcing, thinking about program synthesis as a perspective
  • Group discussions were especially valuable. We debated (eg) whether regularization can account for successful generalization, and I learned about several new papers on this.
  • Clarified a lot of emerging thoughts I was having
  • Zach’s blogpost was the most useful reading and perhaps the Platonic representation paper though I disagree with many aspects of it and think it’s counterproductive, good to have gone through it.
  • First reading was great, I also thought the Platonic representations was also good, just because it referenced a lot of results (e.g. stitching together model representations) that were useful pieces of knowledge to have
  • Learning that we don’t yet know exactly how models generalize and that it’s debatable on if they are actually learning or not.
  • Wow so many, the idea of program synthesis, and different representations of that idea: soft inductive bias. How this explain curse of dimensionality, but we don't have low-level mech interp proof -- we need to utilize indirect evidence. Zach's post presented 3 paradox, while the soft inductive bias paper presented other technical underground for this.
Would improve8
  • More breaks (in the afternoon), more structured post-reading discussions, and maybe more targeted reading (i.e. telling us which sections to focus on in the papers)
  • Just the long reading does not feel very effective
  • time efficiency and purpose of activities: it is not always obvious what exactly the take-away goals are supposed to be, and the last two days it has not been obvious whether or not those were achieved. Using Claude makes the summary process easier, but knowing what the instructor wants us to know would be helpful. Sometimes exercises (math or code) are good and sometimes discussion is useful, however individual groups can have an imbalance of discussion content and focus that misses the goals unless a TA provides direction. We spent much of the day repeating concepts from yesterday and some of the discussion didn't really go anywhere.
  • A bit more variety would be nice, as opposed to just reading. But I appreciate that that is difficult when we just have to get familiar with a bunch of ideas.
  • Probably the main blog post is a bit too long for this format, so probably a more condensed version would be ideal for this
  • I think getting people to talk in small groups works better than the big group discussion but it would be ideal to keep mixing up the groups. The big group discussions could also be very useful too but ideally there would be some instructor involved to ask people who don’t speak out much what they think. Currently group discussions are too dominated by the same three people (including me btw, I’ll try to be more mindful to give others space to speak moving forward) which seems detrimental to those who are by disposition more quiet and it would be nice to have the instructor now and then specifically ask those quotes people to share their thoughts. Finally, sometimes when the group discussions do get too dominated by the same people (again, I am partially to blame here and will try to do better) to the point where it is eating into time for the readings or other discussions, it would be good to have the instructor move things forward more reliably. This is not just directed at Zach, I think in general the instructors are very kind and patient with conversations going off-topic but I wouldn’t think it’s a bad thing to not hesitate moving things along earlier.
  • More time to read, skimming the papers made me feel I only learn the surface ideas which made conversations harder. Also more time for conversation.
  • 2. Small group discussion is ok but seemed too repetitive if we do it for every reading. Maybe after 2 readings => 1 group discussion? 3. can we have more rest pls? 4. can we have Claude Max account pls? 5. some creative ideas: after ~2-3 readings and it's group discussion, have everyone share 1-minute their understanding of the materials. Can employ some techniques like debate, rapid question, or some working together to prove a connected theorem to aid our reflection. Also, the instructor should prepare some guiding prompts (like what Leon did on Day 1).
Singular Learning Theory Sat 11 Apr 12 responses · energy 3.3/5
Content 4.3 of 5
Teaching and format 4.2 of 5
Difficulty match 3.3 of 5
Energy 3.3 of 5
Read what they wrote32
Worked well11
  • Notes were great, presentation was great and very glad we worked through exercises
  • It was nice to get some hands-on exercises
  • lecture notes were so good!
  • Matt and Kai were incredibly lucid and energetic and solicitous teachers; if l were in a semester-long course, I would be thrilled to discover they were the TAs.
  • Initial lecture, exercise and discussion format - all very good! (And this presentation is massively better than what was available online around 2 years ago)
  • The lecture notes provide a great overview and interesting exercises.
  • Self Study at my own pace
  • Exercises integrated with notes was great
  • Excellent day - great, organized mix of lectures, reading, problems.
  • Loved having multiple TA's
  • Mixture of lecture and exercises, reviewing the exercises/readings, minimal unsupervised learning, different instructors chiming in
What they took away12
  • The insight that the same model parametrised differently can have a different loss landscape geometry depending on degeneracy in the parameter-function map.
  • I now understand better how the posterior distribution shifts from higher-loss-but-lower-complexity to lower-loss-but-higher-complexity as the number of training samples grow
  • volume scaling definition
  • Links to sources to read later
  • Good coverage of the ‘singular’ definition and examples for this
  • The way in which the free energy can be used to explain the balance between complexity and fit (in the Bayesian setting).
  • How neural networks generalize; Internal Model selection.
  • There's predictable ways that parameters change and phase changes make them less predictable.
  • (I was only partially engaged, since I'm very familiar with this material, but was still useful to listen in on the discussions)
  • How all the different ingredients of SLT fit together, and what assumptions were being made in each place. Also, identifying how SLT may help (or not) in alignment research.
  • Just the format was great, having explainers, presentations and very useful exercises
  • Derivation of free energy and llc calculations
Would improve7
  • Perhaps tell participants not to necessarily work through exercises in detail given the time constraints but balance between working on the exercise and then looking it up/asking Claude
  • Maybe make more links with actual AI Safety stuff
  • It was too much too fast for me. At some point I stopped trying to follow along or do the exercises and reverted to reading the document from the beginning at my own pace. I wish I could have been open about this and gone out somewhere quieter.
  • Some exercises later in the day were a bit too easy, so I think there would be scope to push a bit deeper in some exercises, and to make the link when introducing probability distributions clearer. I think understanding the basic definitions deeply is quite central, and I have this to a reasonable extent, but suspect a bit more on this might be possible for complete clarity
  • 1. I want to read counterargument or adjacent-varient theories to know how confident I should be about the likelihood of this theory and how strongly I should pursue it. 2. Implications of SLT to capability scaling or alignment: exactly how does SLT influence mechinterp/devinterp/eval practices? Exactly how do knowing this inform our understanding of current bottlenecks arguments? How popular is this theory outside of AI safety theory? How can we translate this theory to practice? 3. We spent 90% of our time on math proof, and 10% on overall lecture (which still is math-oriented); we would have more impact and solidify our knowledge more if more time is spent on the above points. As participants backgrounds and next goals are broad, spending an exorbitant amount of time on math may not be fruitful endeavor for everyone. Could we have dual-track: splitting groups into two streams; or just simply do 50/50 (50% on math and 50% on discussing the above points).
  • I would love to get some more useful intuitions and visualisations in the notes or on the slides!
  • Spend a little more time on the preliminary information first, but we came back to it later (this worked but could be more intuitive the other way around)
Other thoughts2
  • Overall great day! A lot of value
  • This felt like the best day in terms of organization and material. Even though there was a lot of new information, the pace, balance of exercises, reading and lecture, and breaks was very effective.
Training Dynamics Sun 12 Apr, Guillaume Corlouer 13 responses · energy 3.6/5
Content 4.5 of 5
Teaching and format 4.0 of 5
Difficulty match 3.3 of 5
Energy 3.6 of 5
Read what they wrote33
Worked well12
  • I liked the freedom to choice between readings based on interest (at the end of the day).
  • I am in awe of all of these humble, knowledgeable, enthusiastic teachers, specialised in each topic. How did you find them? What are you paying them? There must be some dark magic afoot. This bit may sound silly, but I also felt much more comfortable with a lower body density in the room.
  • Teaching format again worked well, lectures conveyed the meaning and intuitions fairly well - thank you!
  • Nice exercices
  • Lectures were really good
  • The presentations were great and the exercises useful
  • The exercises to illustrate regimes in training dynamics were great.
  • We had more time to go deeper into the readings which really helped to learn a lot about the topic.
  • Lectures were very good and the format was also fine
  • Studying by myself at my own pace. I managed to go a lot deeper than we managed to in the lecture/in person session, even though I spend much less time, since I was able to basically skip what felt very clear to me based on my background and focus on the problems where I just completely/partially lacked the background.
  • Exercises were representative of the work to put the paper, and presentation
  • Liked the presentation today, the format was digestible. I did find some parts hard but it was nonetheless helpful
What they took away12
  • That grokking might be related to a change between rich and lazy training regimes.
  • The difference between strict and non-strict saddle points
  • Understanding the lazy vs rich perspective better, where this was reinforced through the first lecture, exercises and final paper reading
  • Gradient flow and implicit regularization
  • Usefulness of SVD in modelling
  • It was extremely useful to have a non-SLT perspective about the implicit regularisation story, particularly having a dynamical one.
  • What parameters can control different regimes in training dynamics and grokking.
  • Grokking, that was quite surprising and interesting to debate with others.
  • All of the Saxe dynamics exercises were incredibly useful. Very good choices of topics today and the lecturer was very clear.
  • It's exponentially more likely that you'll have saddle points in higher dimensional NN spaces and that predicts that you are almost always just in a "slow/saddle" phase and it's only a matter of time until you move to a different position if your feature space is very highly dimensional.
  • analytically tractable models are hard
  • Understanding divergence, and how/why some training dynamics can alter the model
Would improve7
  • Sometimes questions from the audience lead to discussions that lacked clear focus and relation with the topic. Maybe these questions could have been prevented by starting with a more precise demarcation of what we can and cannot hope to learn from studying training dynamics (in the short term), so that people know what to expect.
  • I think the lectures could have been a bit more concise, both on the side of what was said, as well as less participant discussion. (As a very minor point, Guillaume could speak a little more quietly in a relatively small room like this)
  • I would benefit from more elaboration on how this picture is distinct from SLT and ways in which it focuses on a more general set of possible explanations
  • I would have liked to go over the math exercises in a more rigorous way where we show the solutions on the whiteboard for example.
  • Give people spaces to work quietly at their own pace and implement reminders for them so they can try out whether the next session is useful for them bcs it assumes the amount of context/background they have. This also adds time limits, which makes it much less likely for independent students to procrastinate.
  • shorter blocks of exercises/reading between check ins
  • I personally like a quick review of the basics, but given everyone is very advanced, I don't mind there is none
Other thoughts2
  • Thank you! (I've realised my feedback forms are likely not expressing the gratitude I feel where I'm writing them more plainly)
  • I love Iliad ♥️
Reinforcement Learning Tue 14 Apr, David Quarel 15 responses · energy 3.8/5
Content 4.2 of 5
Teaching and format 4.1 of 5
Difficulty match 2.5 of 5
Energy 3.8 of 5
Read what they wrote37
Worked well12
  • The exercises were well-structured and informative. I also liked to have sufficient time to work on our own.
  • The exercises (I did the practical/coding route) were instructive and fun.
  • Again, I'm very grateful to have the cache of materials to return to later.
  • Working with Aiden
  • Very well rehearsed speedrun lecture, not getting caught up on inane questions, coding/practical exercises - im sure the proofs are useful but actually playing with parameters and debugging misunderstandings/pair discussion was the most useful
  • The presentation was a very useful refresher on tabular RL
  • Enjoyed the lecture, really liked the exercises (completed theory, started empirical)
  • Interesting exercises
  • The colab notebook was excellent (relevent info was reviewed, problems were clear + useful)
  • the lecturing content covered the exercise content well, all of it gave useful anchor for what the math track illustrated and the math wasn't too divergent from the slides.
  • The lecturer was great at explaining complex topics in clear and easy to understand ways. Also the format of having one 45-min lecture and then the whole day to do exercises with the choice of tracks worked perfectly as I could choose what I was most interested in and could work on the exercises (in a pair) at my own pace and speed of understanding.
  • Dynamic and efficient presentation to start the day !
What they took away12
  • I think I got a better understanding of why RL convergence results hold, and how this depends on e.g. the learning rate and the policy learning algorithm.
  • How basic RL algorithms work, with some intuition for the parameters involved and how to improve them.
  • I had seen much of the content before (although approaching familiar things with "the beginner's mind" is a very wise thing to do, in ~Buddhist philosophy!), so having the latitude to run variations on the experiments in the colab (instead of just straight implementations of the instructions) was welcome.
  • The difference between Q learning and Value Learning
  • scheme for optimizing agent decision-making and how to implement using gymnasium
  • That we don’t have a nice elegant expression for a stopping criterion for Q value functions as we do for value functions (only PAC bounds).
  • Understanding of Q-learning for RL foundations
  • Q-learning vs traditional RL
  • I think the fact that the Bellman operator being a contraction map (which seemed key in many of the derivations in the exercises) was a good fact.
  • The generality of the conditions on which the theory is built (esp. thm. 6.1 in the exercises) was pretty astonishing, exposing the wide range of tasks with potential RL applications as solutions. Hope to learn more about the limits to it, though
  • Just by changing epsilon, essentially one parameter, we can design solutions to "be more careful".
  • maybe temporal difference / eligibility traces
Would improve11
  • Maybe we could have had also briefly covered some other RL techniques, such as the ones used for LLMs. We did cover this already somewhat during the MechInterp module, but it would be nice to revisit now that we have more background on RL.
  • The initial lecture was quite fast. Inserting another short lecture/discussion right after lunch might have been good.
  • I think the biggest obstacle to my progress/enjoyment was my own exhaustion/saturation, after a different topic every day for the last while. That might or might not be something for Iliad to improve somehow (see below), but it certainly isn't David's fault! I normally find it difficult when the lecturer speaks very fast by default and makes it the listener's job to cry out for a halt when they can't bear it, instead of proceeding gently by default and reading the room, but I had no trouble following David. He came across as incisive, with delicious bursts of humour in what could have been a dry, stale rerun of old material (old historically relative to sota, and old to David as a learning human).
  • Idk
  • better environment than upstairs (initially, we moved downstairs later) for pair programming/discussion
  • Spend a bit more time on the initial lecture before separating into theory/empirical.
  • The exercises were too long to fully make it through in one day and in the end I think some mix between the theory and practical might have been a better mix. Equally, we could also have just assessed that for ourselves.
  • A little more lecture or q and a
  • Every once in a while is fine, but I wouldn't want one long block of exercises for every afternoon/session
  • During the exercises most if not all people seemed to have questions that arose at specific points of the exercises and I believe the answers to those brought some conceptual clarity to the teaching content, without specifically pointing out where. I think taking time at the end or at regular intervals to gather for answering questions, and then recapping the course material could help deepen the knowledge obtained this way. (I had the opportunity to listen half-attentively to the answers and that felt helpful, anyway.)
  • I would have liked to see some live coding for some of the answers of the notebook to see which parts or edge cases we could have missed when filling our solution in and maybe have a description of the intuition to draw to implement the equations into the code.
Other thoughts2
  • Could u order Chinese food for next weekend pls
  • I was on the math track; I didn't fill these forms up to this point, favoring some freeform personal feedback notes (to be shared), starting now I'll make an effort to write in both!
Idealized Agency Wed 15 Apr, Fernando Rosas 10 responses · energy 3.8/5
Content 3.5 of 5
Teaching and format 3.9 of 5
Difficulty match 3.1 of 5
Energy 3.8 of 5
Read what they wrote24
Worked well9
  • The first part of the day was my favourite part of the whole programme! The lecture was clear and engaging, and then I got to reread all the definitions and do the exercises at my own pace, understanding everything I was doing, occasionally (I think twice?) asking questions, without being pressured to speed up and skip things in order to be ready to rejoin the collective train of thought within 20 mins or to pair with someone who didn't want to go as slowly. This particular content, about AIXI, was/is not a priority for me in the context of my life outside of Iliad, but I expect that a month from now, or three months now, I will remember that content better than anything else I've learned in this program so far, because I was allowed to take the time to understand it.
  • Both lecturers were very clear, and I enjoyed the morning worksheet/exercises.
  • Quick lecture and recap of material
  • The first part on AIXI was taught efficiently.
  • Diverse combo of lectures, problems, discussion
  • I liked the balance between lectures and working on the exercises
  • David's lecture was good, exercises are nice for solidifying well, and Fernando's lecture conveyed the ideas of the area of studying the topic nicely
  • Good presentations !
  • Lecture from Fernando today was really great
What they took away9
  • in the past talking about Bayesian paradigms has felt a bit like switching briefly into a foreign language I haven't formally studied, knowing I won't sound fluent, and just hoping to get through the transaction without any mission-critical misunderstandings. I think I will sound more casually fluent after the unrushed time I spent today, carefully attending to every subscript and superscript on every character in every formula for several hours.
  • The AIXI framework + it's self optimizing property
  • basics of aixi and relation to previous policy learning programs
  • Fernando’s precise formulation of the assumptions behind standard reward models.
  • Fernando’s lecture was extremely insightful and useful
  • Appreciation that the RL from yesterday can meaningfully be extended in the theory, and that studying preferences can tell us a bit about reward functions
  • AIXI and preference representations
  • Preferences under certain assumptions are generated by a utility function.
  • VnM theorem
Would improve6
  • I wish Francisco's two-hour lecture in the middle of the afternoon had been presented as optional. After carefully loading my head with all the unfamiliar notation from the morning, it was a shame to scramble it again. I had trouble staying awake.
  • material was interesting but didn't seem particularly relevant/important as a direction of study
  • The lecture from the second part, on preference axioms that "justify" RL, could have been more engaging. The material was still interesting though.
  • We didn’t really have time to work on Fernando’s exercises, that would have been very useful.
  • There wasn't really enough time on David's exercises to get far in, so a bit more time would have been nice to work beyond what was largely the technical setup, and I think Fernando's lecture could have been faster and with less audience interruption to convey this better
  • The exercises were too long.
Computational Mechanics Fri 17 Apr, Xavier Poncini 13 responses · energy 3.8/5
Content 3.9 of 5
Teaching and format 4.2 of 5
Difficulty match 3.0 of 5
Energy 3.8 of 5
Read what they wrote35
Worked well13
  • Well organized mini-sections with realistic scope, exercises and breaks
  • Very good presentation and super interesting topic
  • Really nice hand-written notes and instructor was attentive.
  • The coding exercises were nice.
  • The activities were varied but well enough scoped such that neither low level technical exercises nor high level theoretical excursions felt disconnected of the core topic.
  • The detailed schedule provided impetus and kept us on track. I also liked the diversity of the modules (lectures, exercises, reading, etc.).
  • Lecturing quality was very good and fluent
  • The fractal visualisations were obviously beautiful! I liked having a full 30 mins for break in the afternoon.
  • Great mix of theory, exercises, and discussions, focused on developing intuition about a small example.
  • The morning with slides and then the afternoon just for readings and exercises worked really well.
  • Good balance between lectures and exercises and papers
  • Atomic and very mathematically interesting lectures, frequent exercises, nice paper for reading at the end, being able to speak to the Simplex founders
  • Xavier did a great job + the lecture slides and exercises were all very thoughtful
What they took away13
  • Definitions of ghmm's and hmm's, and how nns can internalize them
  • The difference between belief states and next-token predictions seems super relevant
  • What is an optimal predictor in comp-mech? How do optimal predictor represent their internal representations? How does this matter to MechInterp? Potentially relating this to my research on "understanding-based eval"
  • HMM embeddings also show up in transformers.
  • Bayesian updating in NNs has been framed in the course so far as a compelling but not well supported hypothesis, but since it's implicit in belief state update, the latter's presence in transformers seems indirect but strong evidence that it emerges.
  • I now understand the Simplex logo :)
  • Q and A with simplex founders
  • The intro placing instrumental goals in the context of AI safety and AI progress, with the scale-free concept, was memorable and may change the way I think about other bodies of work besides this one.
  • Another structured, mathematical way to think about how reasoning may be encoded in neural networks. Good hypotheses to explore.
  • Understanding how quickly a low number of interactions can become messy and difficult to interpret.
  • The papers “transformers represent belief state geometry in their residual stream” and “neural networks leverage nominally quantum and post-quantum representations” were extremely insightful and useful.
  • Found the paper interesting, understanding the HMM theory
  • The GHMM and "quantum" representation stuff I wasn't very familiar with, so it was useful to have a block of time to go through that. Not covered in detail, but Probabilistic context free grammar were mentioned in passing, and reading definitions on my own was useful
Would improve8
  • Slightly more signposted exercises, but the open endedness was nice
  • The time management should have been better. I would have appreciated having more time for the exercises.
  • Some topics I've found unsatisfactory info on: theoretical interpretations of the emergence of the belief state geometry representation, its limits in the data generating process complexity given limited learning model dimensions (which was inquired about during the day), ablations (or some more sophisticated architectural variations) that are sufficient/necessary to break or weaken the belief state representation, whether belief states emerge only when trained on clearly HMMs or they are useful approximations of unknown or not clearly Markovian data generators, probably more. The topic seems to feed well into motivating RL optimality and universal prior related curriculum. To connect them, though, a more analytic or more quantitative understanding of how deep-learning in particular enables belief states to be represented should be instructive. It may actually have come up during the DL intro but slipped my attention. (I'm wondering if upon revisiting, or even first hearing about it later, today's material would be a useful anchor for capturing it? It seems likely.)
  • The schedule was a bit too ambitious: we ended up not reaching the last module. But as I said, that did keep us going so maybe that's not entirely a bad thing.
  • The other teachers gave us colab scaffolding, with access to solutions, when they wanted us to code something unfamiliar in 20 mins. Some of them also gave us clear Plan B’s, so if a few of us couldn’t keep pace with their vision for the day, we could fall back to doing/reading something easier/slower that was still relevant and useful.
  • I think some live coding of the theoretical concepts into visual applications like what we saw in the exercises would have been useful.
  • It would be great to hear a bit more at the start about the overlap between mechinterp and compmech so that it’s clear how these fields are connected but different or rather that compmech is a subfield of mechinterp
  • I think the timings were slightly squeezed in some places (the initial programming task, and the presentation), and the lecturer could be a bit quieter where the room is fairly small. I think highlighting the meaning of the non-invertible map A would be good, because it took me a bit of time to properly understand that the states were not redundant because the state would enable several token prediction, whereas A non-invertible means that for single token prediction the belief state doesn’t fully matter (in the suitable way). I think it took me a while to believe in the idea of studying HMMs as showing anything useful, so maybe highlighting that they’re fairly broad might be good (but maybe this was done). I never properly understood GHMMs, but maybe it’s not important for a 1 day exploration. I think the exercises maybe could cover the theory a bit more, and having a handout/notes in LaTeX with these would be really nice (if it’s worth the time to create).
Other thoughts1
  • Thanks! I found it very interesting
Data Attribution Sat 18 Apr, Louis Jaburi 14 responses · energy 3.5/5
Content 4.1 of 5
Teaching and format 3.9 of 5
Difficulty match 3.1 of 5
Energy 3.5 of 5
Read what they wrote36
Worked well13
  • People made jokes in a collegial mood
  • I think I have a reasonable overview of the research directions in data attribution now, and the main methods for this. Seeing the results of papers in the class was very nice.
  • 1. The QR code for questions were such a good idea. although it was underutilized I think we should keep this concept for other models moving forward. 2. The instructor showing relevant papers was very needed. 3. Good context behind what's the use of this research agenda and its relevance to both AI safety and other concepts we've built upon (SLT).
  • Lot of time for questions, lecture notes were extremely good
  • Good presentations
  • The lecture notes are great, and the number of topics/sections was appropriate for the amount of time we had.
  • The notes are informative and useful
  • The morning of lectures and afternoon for reading and exercise worked well. Also being told to use Claude to work through some of the "less interesting maths" was quite useful, having the validation that we can just move on if we get the idea is nice.
  • Lecture is very good and setting times ppl have to be back on discord made class flow better than normal
  • The notes were extremely detailed and it was useful to see everything spelled out so thoroughly
  • Good notes (great intro to important parts of the topic; technically a bit dense but good explanations throughout) and group discussions
  • The walk! I wasn't expecting it would benefit me, since i didn't consider energy level to be a problem in the afternoons, rather a diffusion of focus, and it ended up helping with that. Of course it's just 1 experiment, so no conclusions yet :) Also the group cohesion went OK, i felt we were more dispersed on other days, which i didn't enjoy -- but there's still room for improvement imho...
  • Breaks, mix of reading and exercises, regular and scheduled check ins
What they took away13
  • The connection to optimal transport was unexpected and cool.
  • The idea of doing unfolding with automatic differentiation is really nice, and the ideas for future research directions are nice (e.g. nothing known in RL)
  • Unrolling, influence function, bayesian influence function
  • Just generally getting an idea of what people are working on in data attribution! It was almost entirely new but super useful
  • The high-level considerations of data attribution
  • The general framework for data attribution: assigning weights to the datapoints, computing a relevant model configuration/checkpoint, evaluating for some property of interest, and then backpropagate to measure the influence of each datapoint. It was interesting to see different examples of this.
  • The unrolling seems interesting and useful, especially MAGIC in that it provides information about the causal impact of a datapoint throughout training
  • How we can skip lots of computation by looking directly at the weights and evaluating them.
  • Learning about meta gradient descent this was huge for me really interesting
  • The fact that classical IF are approximating proximal Bregman response function and not leave-one-out loss was very nice to see. Also the unrolling content was good + something I hadn't seen before.
  • Current frontier of the field of data attribution
  • The entire idea of influence functions was unfamiliar to me, although i have heard the term, this was an introduction to me. It was inspiring, albeit partly due to dissatisfaction with the underdeveloped theory (and that's probably the way it should be).
  • Data attribution is messy, there are some approaches in development of varying rigorous justification, there are some industry standard/practical methods
Would improve10
  • I fell behind and never caught up. I felt like I didn’t have the prereqs for this vision (pace, scope) of the day, and my time would have been better spent abandoning the curriculum and re-installing those dependencies than lumbering in the wake of the instructor’s train of thought.
  • I think I found the notes relatively hard to follow, where I think I'm not especially strong on multivariable differentiation intuition, and I think the notes were some intermediate of being very precise and being an overview, so probably more clearly defining everything said, or being clear where deliberately not, would help. It feels like the overall ideas are quite straightforward, then the implementation or approximation methods are much less so, and possibly the balance of time on the methods vs experiments + results from it was too much of the former.
  • 1. Should highlight direct implication/most relevant results from research papers, and any remaining bottleneck questions left unfulfilled with current research to guide research directions. 2. What effect does this have outside of AI safety? Would digesting on this accelerate any capability potential work?
  • Im not sure that the matrix calculus details are useful.
  • We spent far too much time on influence functions and BIS which are in fact not very useful thw former because they don’t apply to NNs and the latter because SGLD doesn’t give reliable estimates of the quantity of interest.
  • I would add more regular small breaks in the afternoon.
  • It was alright for me but I think there are a few people today for which the mathematical density of the notes was probably prohibitive based on my experience with others in the past couple of weeks?
  • A coding exercise where we calculate influence functions in a trained model might be nice to include.
  • Q n A seemed divergent. It is a highly subjective opinion, that's true. I may think so because the preparations (planned timeslot and askqueue) made it seem like it would be somehow more structured, focused and/or better scoped; also i often try to listen in on other's questions and discussions with the TA outside the class setting and today this wasn't feasible in multiple cases. In any case, today i became mindful that synthesizing different lines of inquiry that different participants follow can be an aspect in which some days' teaching and format can work better or worse. (Now i'll see this as an even bigger challenge for tomorrow's "cross-pollination sessions" -- although it may turn out to be irrelevant.)
  • Document could use more refining/some more literature analysis/practical exercises
Agent Foundations Sun 19 Apr, Sung En (Daniel) Chiang 15 responses · energy 3.6/5
Content 3.4 of 5
Teaching and format 3.7 of 5 n = 14
Difficulty match 3.2 of 5 n = 14
Energy 3.6 of 5
Read what they wrote36
Worked well12
  • Interesting discussions and readings
  • The format was good (and I enjoyed the forcing of small group discussions from people who read different topics), and the short lectures from different researchers was interesting.
  • The pre-reading combined with group sessions lead to interesting discussions. But I am not advocating for having pre-reading for all modules, that would be too much.
  • The option to chat in groups was great would have liked more time for that.
  • Guest lecturers were good, cross pollination is a really nice format that people seemed to benefit from. The exercises have so many valuable remarks they are close to supplementary lecture notes.
  • teaching format, early-sent materials, guest lectures, and a nice mix of lecture, exercises, discussions, and guest lecture makes today feel much better and more balanced.
  • The private small groups with advance reading were great! More like that, please! I think because we’ve known and liked Daniel for two weeks, I felt invested in wanting the day to go well for him, so I even if I was tired, some subconscious relational imperative kept me more engaged with him than the one-day teachers?
  • Talk with Aram was nice, group discussion worked reasonably well
  • I think the topic selection was important, i had been aware of some agent foundations concepts and also the somewhat dismissive attitude towards the area. (I got the impression from some participants that they've come to share that sentiment to some extent.) My appreciation of the questions increased and i at least understood how they relate to each other and AIS in general.
  • The mixture of lecture, exercises, readings, and discussions was nice. I enjoyed having different formats to learn that switched about every 90 minutes and it kept me more engaged.
  • Thorough introduction to the basic concepts, explored by discussions.
  • The discussions were really good, I enjoyed the conceptual nature of the lectures, with the more theoretical maths based exercises.
What they took away12
  • Decision theories
  • Thinking of mutual information between an agent and system as a resource for optimization was an interesting perspective.
  • I now have a better sense of all the modelling difficulties introduced by embedding agents in a larger world.
  • The lecture about studying the physical origins of causality and choice was extremely useful and insightful.
  • Premise of algorithmic thermodynamics
  • decision theory
  • Newcomb’s paradox splitting the world into one-boxers and both-boxers, with both groups thinking their position is obvious!
  • CFT vs FDT idea (and from non-normative perspective), embedded agency idea
  • That the confusion and problems that i've struggled with in connection to the arrow of time, causality, entropy etc. are shared and not understood with a consensus on the wide AF community. It gave me hopes that i have a specific contribution to make currently.
  • Agents are part of the world rather than having an entire understanding of it.
  • Open problems in decision theory and how they may practically relate to AGI.
  • Recapped on different reward modelling systems. Learnt about lobs theorem (and the theorem it's based on)
Would improve11
  • I found the talk by Cole not great, I felt he was a bit rambling and it was hard to follow. That's a comment about the presentation style, not about the content (which I think was good).
  • I think I'm still a bit skeptical that some of the more abstract topics we've covered in Illiad will be concretely useful for AI safety on reasonable timescales, but perhaps this is more personal taste than fundamental objections.
  • More realistic scope/focus
  • The first two lectures could have done with a break. The second lecture also had some unnecessary overlap with the first and the idealized agency module.
  • The lecture about AIXI really wasn’t useful to me and seemed to not really be an improvement. I would have preferred a lecture on one of the other areas of agent foundations. I liked Daniel’s lecture but the text on the slides was too dense.
  • So much content a lot of other people didn’t get to the exercises
  • please don't order fast food anymore, instead can we have more Asian food please please!! Also can we also go to Asian restaurants after class as well. I could barely ate anything yesterday and today due to how the food choice is.
  • Daniel seemed nervous. He could have talked slower. The fundamental reading was pretty fully redundant with his talk? I didn’t see the need for both of those. (The pre-reading was non-redundant and valuable.) Cole Wyeth’s talk was perplexingly bad. If he was just extremely nervous (to the point of tripping over his words and repeating himself so much that he ran out time for any non-redundant material before the end of his hour-long slot), I would like to extend my sympathy and hope that getting practice helps him feel better over time. If he just wasn’t unprepared, I wonder if we could tactfully require that he rehearse before coming back.
  • I think the initial lecture and Cole’s lecture could have been more focused and clear in scope - I found I didn’t understand that much of the first because it was perhaps too ambitious in scope, and I would have learnt more clarity from a more restricted and focused session, and I think somewhat similarly for Cole’s lecture, I found it somewhat unfocused so struggled to get clear points out of it
  • In the material: the acknowledged existing challenge to reconcile many opinionated and partly contradictory research agendas may be the root cause of some unfavorable views on the field, and a more positive outlook could be conveyed to everyone with evidence that "integration to real science" is ongoing. In the teaching format: I felt that the afternoon was divided into too many segments and assigned discussion/exercises did not get enough dedicated time to work on them.
  • I would add some content on "where are we today" to explain where the most modern agents research is and what it can currently do.
Other thoughts1
  • Generally it would be good to get a list of "what we should take away from this day/content" at the end of the day, or a set of questions to answer to test what we have gotten from it. This is not specific feedback to this day, but more in general for the course!
Alignment in Practice Tue 21 Apr, Evžen Wybitul 13 responses · energy 3.9/5
Content 4.5 of 5
Teaching and format 4.4 of 5
Difficulty match 2.8 of 5
Energy 3.9 of 5
Read what they wrote34
Worked well13
  • Excellent format, teacher, and content today! Everything was great, and I especially enjoyed the red team/blue team activity!!
  • Group discussions and especially the debate - with genuine pressure to think about problems and participate (because answers would be shared with the class)
  • Evzen is just really good at presenting and guiding people. The discussions were super good, we covered so much ground. It was very energizing and just a very useful day overall.
  • Debate format I found so much fun but he is such a good lecturer that maybe it’s better to just lecture per unit time I’m not sure. Regardless the debate was so much fun I feel like it improved overall experience of Iliad for some people including me
  • The content was very interesting and with examples on SOTA models and real world implications. The debate format and group discussions kept it very engaging.
  • Great format, great presenter (with a good balance between letting us discuss and bringing us back on topic/schedule), very relevant content. I especially enjoyed the Red-teaming exercise (debate).
  • It was easy to follow, much gentler and more forgiving than the more maths-intensive days
  • Teaching format and regular discussion + debate game were really nice
  • The pacing of breakout discussions, as well as the content helped keep up engagement and thinking, respectively, this made today stand out very much as a positive learning experience.
  • Quick discussions & brainstorming, review of paper results, relating discussion to practice, debate exercise worked very well
  • Lots of papers and examples.
  • Very useful to get this overview of what is actually being done in practice, and the difficulties of the related problems.
  • The red-teaming exercise and the discussion sessions interspersed with the lectures were fun, and were good for preserving focus.
What they took away12
  • Learned lots of different jailbreaking techniques that were super interesting!
  • Strategies for safety and jailbreaking… as with many days, the material has been largely new to me, and the overview today was very high quality. From a research perspective, today set out the ultimate practical goals that I feel I need to keep in mind when investigating more mathematical aspects.
  • It was good to get a complete overview of alignment practices! And to try ourselves to build a safety case!
  • There was a lot of bits of knowledge that were very current about things like auditing success rates that I thought were extremely valuable and useful for me to decide what is important for me to work on
  • That post-training is where the persona building happens for LLMs but world understanding is mostly in pre-training.
  • 1. The general structure/pipeline of alignment in practice. 2. The general takeaway that SOTA alignment methods are not that complicated (and not that robust).
  • I am converging on a theory that this programme may have been designed by and for extroverts with ADHD, and the main benefit for me on many days will be the beautifully calibrated reading lists provided by each instructor, which I can take home and do at my own pace.
  • Intuitions for the whole process, more feeling of the 'spirit' of trying to red-team
  • There is much in the material to be let sunk in, and many things to remember, which will require an eventual revisit. Latent around it was a new recognition (to me at least) that there are numerous known threat cases being studied proactively by labs, and they take their discovery seriously at least at the level of allocating talent and resources to it (of course the amount of information and interventions still seems alarmingly low but the neglectedness is likely not as bad as, say, 3 yrs ago).
  • Alignment in practice is hard but takes many different forms with trade off, understanding of literature relating to jailbreak and vulnerability at different stages of training
  • Persona characterisation of models
  • Gradient routing seems really interesting a tractable compromise on getting a more explainable model without sacrificing performance too much.
Would improve8
  • Bit too much material for one day.
  • Honestly it is just super good, I think more people should do this setup with Evzen it is just really informative.
  • The conversations could have been slightly longer to let people have more in-depth conversations.
  • I did not feel like we had enough time or exposure to develop sincere opinions for the debate or the small discussion groups. They felt fake and forced.
  • I think we could've gotten more out of the debate if beforehand there was an opportunity to summarize and structure the day's learning, to remember and be inspired by it, and possibly it wouldn't have hurt if some bits of the material were skipped because of that. To be honest, this section of the material could do a good job at introducing currently relevant threats, possibly better so than the actual introductory section. This applies mostly to LLM-related AIS of course, and I view it a valid unanswered question how general this can be made vs how LLM-specific it is -- I think there's something to be learned here about aligning various different, narrow purpose models, which we gestured at with biosec risk, but still remained anchored at experience with Claude...
  • Another day on it with more literature/practical exercises/tests
  • I do not think the pink sticky notes on the board were useful. I did not hate them, I just think they were pointless.
  • Perhaps the constraints on the debate could have been clearer, like for blue team “prioritise defining uplift to bioweapons clearly”, and for judges “only one technique per training section”.
Other thoughts1
  • The lecturer mentioned that the vulnerabilities found by Mythos could also be found by weaker models if provided a good harness. I believe this was in reference to this article: https://aisle.com/blog/ai-cybersecurity-after-mythos-the-jagged-frontier This conclusion is somewhat misleading, as the "harness" involved only showing the models the vulnerable code. > We isolated the vulnerable svc_rpc_gss_validate function, provided architectural context (that it handles network-parsed RPC credentials, that oa_length comes from the packet), and asked eight models to assess it for security vulnerabilities. I.e., they didn't provide the entire codebase. The experimenters used their knowledge about where to look for the vulnerability, and provided this advantage to the models.
Abstractions Wed 22 Apr, Sung En (Daniel) Chiang 10 responses · energy 3.5/5
Content 3.0 of 5
Teaching and format 3.2 of 5
Difficulty match 3.3 of 5
Energy 3.5 of 5
Read what they wrote25
Worked well9
  • The readings were interesting.
  • The exercises were informative, and I like the group discussions where read different topics
  • Good presentation at the beginning
  • Compared to C3 the amount of material (being less) matched better what felt like could fit into a day. I liked the format and could extract more value from the cross-pollination conversations yet the day still seemed intense.
  • Initial presentation
  • Good to get knowledge on some ideas in the field like natural latents
  • It was sunny. Now that we’re 2.5 weeks in, people know each other better, so they seem less inhibited when discussing, as if they have ample evidence to trust that mistakes and disagreements will be handled graciously within this cohort.
  • Good initial lecture, good readings
  • Exercise sheets today were probably the most relevant and informative of the whole of the intensive so far I got the feeling I was learning something very significant in the way I do during an actual maths class. Quite motivated to learn more or contribute to natural abstractions now.
What they took away8
  • That natural latency stays stable between agents with different internal models.
  • I enjoyed the section in [participant]'s writing about people attempting to NN in QFTs
  • The concepts of natural latent and condensation
  • Even with its limitations, the latents framework was interesting to explore. I have encountered it earlier and mostly dismissed it for being too idealistic and far-fetched, and after today i find it significantly more motivated and i can articulate my concerns with it better.
  • abstractions has a long way to go and is missing empirical demonstrations or studies of falsifiability
  • Concept of mediators and refunds was useful
  • I had a nice chat with Claude about the history of the term entropy (and its relationships with KL divergence and mutual information) during the exercise portion of the day.
  • Mathematical formalism for abstraction - and great group discussions on how we might test this formalism in “toy model” AI systems
Would improve8
  • I would add the solutions to the exercises so it is easier to check if I am going in the right direction.
  • Put more real-world motivation or draw more links with concrete things that have been done. I found it was harder to motivate myself compared to other days (although this could also be due to my energy level today and/or random motivation fluctuations).
  • My experience with the exercise activity was mixed. On the one hand not having the solutions, nor much of a summarizing discussion (well, we had some of that but really not much) it felt rather uninformative, similar to some early days (IIRC the deep learning days), with what seemed like math practice exercises and little to take away. It may have had to do with my energy somewhat decreasing towards the end of the day. But on the other hand, this prompted more small-group discussions than earlier, which was an enjoyable experience. Lately the TA involvement in breakout discussions has been fading. This makes me feel insecure sometimes about the direction the conversations go (regardless if i or others are steering it). It would be useful to have more of that!
  • extract falsifiable predictions from john et al. using enhanced interrogation techniques, dial back the scope
  • This area really seems to lack some experimental demonstrations, even for toy models. And I think people were less engaged in the material because of this. The questions of interest need to be constrained by actual examples and it doesn’t seem like this work has been done.
  • When Daniel lectures, he talks very fast. I wish he would speak more slowly. I would have liked it if he summarised the key takeaways at the end of day, taking on some responsibility for connecting the disparate exercises and readings to each other, instead of leaving that up to us. I think the idea to have a day about abstractions could be good, but this curriculum felt hazy and undirected. It was as if someone with a proper plan to teach us got sick, and a substitute threw together a plan at short notice, without more of a goal than to keep us nominally occupied.
  • A final afternoon session bringing everything together would be nice.
  • Optional readings at the end were too hard for allotted time
World Models Fri 24 Apr, Fernando Rosas 11 responses · energy 3.5/5
Content 4.1 of 5
Teaching and format 4.0 of 5
Difficulty match 3.0 of 5
Energy 3.5 of 5
Read what they wrote32
Worked well10
  • Good presentations
  • I really liked this slightly more generous break schedule, combined with the earlier stopping time: I felt present and engaged almost until the end (and with just a slight slump at the end). Fernando has a natural way of drawing his audience into dialogue with him and sharing in simple words what he feels/intuits about the material before and beyond the provable theorems it contains, which also helps a lot.
  • The content was interesting and the lecturing was good. I also appreciated that neural networks were actually mentioned today, instead of just abstract reasoning about math
  • The pace was relaxed. This is not always a desirable thing, but today I appreciated it.
  • The content was really interesting. It was nice to have long breaks in between lectures to actually rest my brain. The multidisciplinary aspect of the talk made it very engaging.
  • I found today's lectures to do the best job at connecting different earlier topics. This is perhaps not surprising given that we have a quite large (and still increasing) ratio of covered curriculum to draw references from, but Fernando worked well on citing or pointing to relevant topics (as well as had many to begin with).
  • Lectures in the afternoon on transductions, hierarchical emergence etc were so interesting and well presented. I really liked having today so close to abstractions day.
  • Enjoyed the lectures, found these a reasonable overview of some nice world modelling ideas and the area
  • A good intro to the idea and ideals of world models, and current methods for course graining decision processes.
  • This is the sweet spot for me between philosophically interesting and being plausibly empirically testable (even if we can only test in contrived environments for now).
What they took away11
  • The concept of computationally closed coarse-graining
  • It was fun to talk to Fernando
  • Negative probabilities! (Or, negative transitions, latent dynamics that are not probabilities.) I hunger for another way to interpret them.
  • Learned what \epsilon machines were! Also, the relationship between symmetries and hierarchical coarse grainings was interesting.
  • I learned another way of formalizing abstractions/coarse-graining, which could be useful for comparison with the Computational Mechanics and Abstractions modules.
  • Understanding JEPA and V-JEPA as I have heard a lot about it in the news and here it was very well explained. The generative simulators Sora and Genie also were really interesting.
  • The epsilon-machine-factorization-computationally-complete-levels correspondence (i'm just hoping i'm getting the terminology right, my attention span was occupied with listening and didn't take notes...)
  • Maybe one of the results about unique smallest transductors, or maybe the initial intuition for the hierarchical stuff.
  • having the 6 axes of world models as a basic guide, seeing the approaches to the diagrams for transducers and the corresponding results for these, clear abstractions for the justification of coarse graining in the computational and information settings, and nice ideas on using symmetry to generate these
  • It was a good reminder to think about scaling and course graining more broadly in training neural networks!
  • The last lecture was extremely useful about hierarchical emergence and the connection with computational mechanics.
Would improve11
  • There were some connections with actual RL/AI systems, but I think the lectures would benefit from having more of that. Also, the initial presentation about the concept of world model was not super clear for me / maybe going in too many different directions or presenting too many different versions of world models.
  • Maybe more exercises instead of only presentations/ more discussions?
  • I didn’t like the first slide deck generated by Claude, after Fernando expressed his own lack of confidence in it. I didn’t feel sure whether I could trust any individual statement or formula I was reading on screen to be true, without asking for confirmation.
  • More solo/group work to parse through things on our own might have been nice.
  • The first (AI-generated) slide deck could have been more visually engaging. The second presentation was much better in this regard. The content also felt somewhat similar to the Computational Mechanics day; maybe parts could be merged, or perhaps the differences could be highlighted more emphatically.
  • The speaker talked a lot about their research and how it connected or developed these concepts and I would have liked to hear more about how you research such concepts and what it may look like.
  • At some points i found the lecture somewhat too fast-paced to think of interesting, informative questions, of which there were fewer today than on many other days. (Looking back, this also applies to Daniel's and David's days.) I'm not sure to what extent this is a conscious design choice, to prevent being stuck at possibly unimportant details, or perhaps due to the day schedule being about 1/5 shorter.
  • Morning lecture on world models was a bit too AI for me. Just lists and sub lists without a ton of substance and definitely not at the level of quality of Fernando’s other lectures (including from the RL day)
  • Probably a few more practical (either maths or coding) exercises would help (particularly if the day is longer than until 4.30)
  • It would have been nice to include some more interactions - eg discussions in groups to come up with good criteria (or examples) of world models, or exercises involving course graining.
  • I approve of getting Claude to generate presentations/do research etc. and it’s good to see presenters do that and say they’ve done that (to establish new norms) but the slides should be checked and reviewed before the lecture is given and this wasn’t the case in the opening lecture today.
Limits of Interpretability Sat 25 Apr, Louis Jaburi 7 responses · energy 2.9/5
Content 4.0 of 5
Teaching and format 3.6 of 5
Difficulty match 3.0 of 5
Energy 2.9 of 5
Read what they wrote21
Worked well7
  • It was useful to think about how we could be fooling ourselves or not using our time well regarding current interpretability methods.
  • Great morning lecture and Python notebook to explore/internalize it.
  • Overall case for finding a better mechanistic approach to alignment than traditional mech interp was really well presented I hope it was compelling to the people who hadn't been privy to this idea before.
  • Good scope on lecture, exercises, and readings.
  • The presentations were great, and the time balance with the notebook exercise and reading session was good.
  • The drawing back to the research the lecturer had done and is doing was really interesting as we could see how this was researched in practice.
  • The amounts of time allotted for the exercises and the readings seemed appropriate.
What they took away7
  • The arguments for caring about faithfulness and compactness were useful and interesting to learn about.
  • Exposure to the idea of formal interpretability certificates. Even though the way they are currently implemented seems hopeless, it’s an interesting concept to aim for in other approaches.
  • Doing some basic compact proofs I think this was pretty useful for my intuition (although it seemed to confuse some other people who couldn't immediately imagine how a proof like this could incorporate facts about the trained model's weights)
  • Balance between cost of proof and strength of bounds, interp is doomed
  • That some people are actually proving bounds for neural networks, but that this is hard to scale. And that there are informal notions of scaling this by weakening from proofs to 'heuristic arguments'.
  • The idea of compact proofs as a whole is a interesting framework.
  • There are backdoors that are NP-hard to find; “interpretability is doomed.”
Would improve7
  • I think the fact that there is no existing consensus on what counts as a heuristic argument could have been established earlier.
  • More could be packed into this day. Perhaps other exercises in mech Interp (and its failure?) to contrast with the proof approach. It’s really enough content for just half a day.
  • Day was obviously a bit short we ended hours early
  • A few more research directions/approaches and open problems outside ARC (if possible)
  • The exercise notebook contains coding exercises in the form of cells with "Solution goes here", but the solution is immediately visible in the next cell (instead of hidden in e.g. a drop-down). However, I think most value comes from the math proofs, so this is not a big problem. In fact, I think the coding exercises could be removed as they are just implementation details.
  • Maybe seeing some of the code that was used to make the lecturer's paper would have been interesting as a concrete example of this research in practice.
  • I have heard comedians say that if a comedian seems nervous, then the audience feels nervous on their behalf and can’t laugh; as a theatre critic reviewing four weeks of one-man shows, I would say that Louis doesn’t seem to enjoy or play in the role of presenter as much as our other teachers do. He crosses his arms and flashes infrequent smiles. He had much less material prepared for this topic than his previous day’s.
Backdoors and Steganography Sun 26 Apr, Stephan Wäldchen 13 responses · energy 3.4/5
Content 4.2 of 5
Teaching and format 4.6 of 5
Difficulty match 2.8 of 5
Energy 3.4 of 5
Read what they wrote30
Worked well11
  • I thought it was very useful to work through the ReLU exercise and present solutions on the board.
  • Discussion of papers with TA's was useful to make sure we got the important ideas
  • Starting out with a game was a great way to be introduced to the core topic and be engaged. The rest of the day being more chill with readings and lectures was enjoyable.
  • The lecturers as a pair were great they were extremely entertaining. Stefan’s(?) concluding talk was great.
  • The steganography game was excellent.
  • The cohort was jolly. Stephan was ebullient. Louis was perhaps more relaxed than when he was flying solo.
  • The game in the morning was fun. A little silly, but got us to think about simple approaches to steganography that LLM’s are also using - it combined nicely with the reading in the morning.
  • Very engaging activities and teachers/content. Readings and exercises were also informative and at the right level of difficulty.
  • The game was fun, and also positioned at the right point in the schedule (i.e. the start of the day).
  • Initial steganograph game, range of activities/exercises
  • Interesting topics and papers, and I enjoyed the varied teaching formats of the day.
What they took away9
  • The backdoor paper was very interesting to me.
  • Backdoors can have serjous security consequences, but implementing them requires white box access to weights and relies on minimal downstream SFT or complex large scale data poisoning for frontier models
  • That chain-of-thought in LLMs can quickly become unreliable and is not a way to actually see what the model is thinking, and that it can be used to pass secret messages to future versions of the model.
  • I think I gained some important mathematical intuition during the contrived backdoor single hidden layer example that is an important thing taken forward.
  • Being elicitable =/= being detectable. Superposition makes it harder to exploit and discover structure.
  • I finally learned why we’re told not to put optimization pressure on COT. Also the Merlin/Morgana setup in the afternoon was interesting for interpretability.
  • I was completely unfamiliar with the concept of steganography, so everything was useful. Thinking through how this can be used for nefarious purposes, and how this is fine in practice was a particularly useful exercise
  • Various nice ideas for this, with the steganography intuitions, backdoor intuitions, and Merlin-Arthur approach
  • Steganography for weight exfiltration. NP-hard backdoors.
Would improve10
  • Perhaps would be good to talk more about the assumptions that need to hold in order to make backdoors impossible ro efficiently discover and how much this maps on to reality.
  • Encryption game was entertaining but idk if it was particularly useful, also exercises seemed a bit contrived and uninformative, but interesting, maybe could have tried to implement or test backdoor techniques in toy models representative of the readings?
  • More whiteboard solutions of the exercises on would be nice.
  • Steganography game was a bit too easy I think a really hard version where the team had to agree on their secret key before they know, not just what the word and question is, but what the medium is like it could be revealed that they have to communicate through some constricted alphabet with some constricted grammar (this teaches something about Schelling points that I think was missing today a little). The other hard version is the obvious one which combines the ones today to write arbitrary messages after paraphrasing.
  • Finish on time.
  • Stephan and Louis both expressed a wish to spend more time improving the exercises they had written quickly, for yesterday and today. The long steganography game at the start of the day may have had non-didactic value for bonding or boosting energy, as a dance break would do, but it certainly wasn’t necessary to get the idea across.
  • The afternoon exercise was a bit too computational… maybe there’s a way to rephrase it in order to emphasize the structure/intuition more?
  • I think I find the 1) reading paper and 2) discuss not the best way to get across the materials, also hard to get an idea of what would be treated today as we did not have a master doc etc.
  • The last presentation was interesting, but it's link with the current state of the field could have perhaps been made more explicit.
  • The mathematical exercise can probably be made more relevant to the backdoor ideas - I don't think it really conveyed that much too me beyond the intuition of relatively small changes allowing fairly high probability of separation (although I still enjoyed just doing gentle maths for a bit :))
Debate Protocols Tue 28 Apr, Stephan Wäldchen 12 responses · energy 2.8/5
Content 3.5 of 5
Teaching and format 3.3 of 5
Difficulty match 3.1 of 5
Energy 2.8 of 5
Read what they wrote26
Worked well9
  • The content was intuitive. The mixture of presentation, reading, talk in the afternoon and finishing with a debate from two AIs was good to keep the energy up and the day engaging.
  • Initial presentation was good (but a little long) and discussion/criticism of literature across different groups
  • The formalisms of the debate protocols
  • Good diversity in teaching formats, and all parts were good !
  • The activities were varied and balanced, the scope seemed narrower than on other days and it likely worked in favor of understanding the material.
  • I liked the debate between Claude and ChatGPT, this was very interesting even if they both failed ultimately
  • The schedule was varied: identifying proof flaws, presentations, collective exercises, reading, and a debate demo. This diversity makes it easier to stay focussed.
  • The afternoon readings about debate in practice for AI safety, AISI talk, and live debate with LLM’s about AI safety (illustrating the protocol) were great.
  • The teaching was good + engaging, and I very much appreciated that in the afternoon we stepped away from toy/exactly solvable examples to proposals to how this will be implemented in practice.
What they took away9
  • Learning about the impartiality and the general role of humans as judges in AI interactions was very interesting.
  • Debate is 🤷‍♀️ interesting framework, but even the researchers can't comment on what works and doesn't so it seems like a hypothetical strategy
  • Cross examination
  • I didn't know about the Debate protocol and I found it super useful to know about this technique and the related criticism.
  • That debate actually has a well-formalized, mathematically rigorous definition, but it's highly non-trivial to map that onto how LLM safety can be improved by debate.
  • In principle I like this idea and find it useful but it seems to rest on many unrealistic assumptions so the most useful thing was to focus on the most specific instances. The most useful thing was this connection to why drilling down to semantic concepts is so hard and how debate in some circumscribed domains could massively benefit this. So in a way Debate seems useful to help us with progress on some fundamental issues that might then help us with safety, as opposed to being directly relevant to safety. In some ways the talk by Marie was more in this direction and that was useful.
  • Different formalization frameworks for debate (cross-examination, prover-estimator), and the idea of obfuscation as a debate strategy.
  • The idea of debate as an effective mechanism for probing safety of models — as well as a useful mechanism for me to get LLM’s to produce good arguments.
  • The afternoon talk with the sketched version of how advocates imagine this being used in practice was good to go through.
Would improve7
  • I would give more papers/article options as I thought hearing people's summarisations of what they read and their opinions (on the ones I had not read) was very insightful.
  • It seems as if this and a few other days could be squished into a single day of various strategies for alignment
  • Maybe describe the goal/motivation/setup more before diving in theoretical stuff.
  • The morning material and method started off seeming thought through and structured. There seemed to be some skepticism in some participants that the idea of debate is either useful in theoretical CS or in AIS in general. A germ of such a discussion started growing before lunch. Afterwards we jumped to the opinions on AIS via debate and I feel like we were unprepared to engage in reading or perceive that content in an informed, objective way due to little time commitment to the concept (and is connection to the morning's material). This made the early afternoon seem a waste of time, discussing things about which we couldn't be well educated. In my opinion laying the ground would also have helped in understanding the afternoon talk. I view the chatbot debate in the end superfluous and uninformative retrospectively.
  • I didn’t find it useful to work through the exercises together so fast as this, I feel like I didn’t learn much from them, would be better to have some time to try it for ourselves. I also found the material from the opening presentation wasn’t so easy to absorb today and the complexity theory details in the end didn’t seem so crucial for understanding debate yet we spent a long time on them, I think the emphasis could be shifted much more towards the question of what debate is useful or important without going so deep into the complexity theory.
  • A sketch of how debate might be used in practice for AI safety could have been useful before introducing the formal complexity-theory-based framework.
  • The more session was a bit confusing. The quantity of formal logic and turning computation in the session was intense. And I’m not sure altogether necessary? Do I need to know about polynomial length in order to implement a debate in practice?
Other thoughts1
  • Note on the low energy score, in case that went into assessing teaching quality, it was on me for not sleeping much the previous night. Also for this reason my focus was low for the late afternoon but I was a bit more energized towards the end.
Outer Alignment and Reward Learning Theory Wed 29 Apr, Leon Lang 9 responses · energy 3.6/5
Content 4.0 of 5
Teaching and format 3.9 of 5
Difficulty match 3.0 of 5
Energy 3.6 of 5
Read what they wrote24
Worked well9
  • The topics were mostly new to me but they were on my radar and felt like i was missing these so the selection was suitable to me (except for Richard's).
  • Even though there were quite a few lectures, they were varied and spaced enough for this not to be a problem in my experience.
  • Good clear presentation at the beginning, scaffolding on exercises
  • Joar’s lecture was really interesting; I appreciate having 2 guest lecturers covering different spaces and perspectives, very interesting to compare their representation of goal and behavior, reward and policy differently. Great choice of speakers.
  • It was nice to have multiple lecturers to get different points of views on the field.
  • Morning lecture and exercises (especially the first one) were great.
  • I enjoyed the varied format of teaching.
  • Guest lectures were very very good Exercises were good but a bit easy relative to the amount of work they took
  • Leon might be my favourite lecturer of the programme. He defaults to a slow, inclusive pace (without needing to be flagged down and begged, which can feel embarrassing), he doesn’t charge shame tax on questions (while also not letting them derail his arc), he laughs a lot like the Dalai Lama does, he knows how to choose specific, clear examples. I hope I “grow up” to have a lecture persona like his someday. I hope he does more of the teaching himself next time. Joar’s content was very cool. I wish we could get him to do a full day.
What they took away9
  • everything about IRL
  • It was interesting to learn about Joar's RL framework, and how it can be used to formalize reward function robustness and misspecification.
  • Joar's talk on STARC metrics and observation dependence for feedback
  • I'm very interested in the research direction of Joar's.
  • The concept of inverse reinforcement learning was really interesting, in the last talk.
  • To explore flexibility in reward models.
  • That RLHF needs very strong assumptions to work
  • Maybe this idea from Richard about how groups of agents combat discounting
  • Partial observability is really bad for RLHF. Also, reward optimisation is our only method for directing AI systems not anchored to or limited by current human preferences.
Would improve6
  • I didn't see the relevance and utility of Richard's talk, and to my understanding a lot of informal assumptions and handwavy explanations went into it, which made it seem professionally lower quality than others. The strong leaning on the utilitarian approach was rather alien and hard to digest for me and I realized that I'm not gaining value from it so I left halfway through, so this opinion might not be representative.
  • The computations in the first exercise were a bit laborious at times. Maybe the number of computations could have been reduced while preserving the educational value (e.g. by removing a state).
  • A bit scattered today between talks, exercises could have been a little more carefully constructed/concise
  • I would give some papers of the speakers to read before to be familiar with their research and have some more insightful questions to ask them, generally I found their presentations really engaging.
  • The material was not quite enough for a full day. I also found the first afternoon talk on agents and decisions distracting - too philosophical for my personal taste, with no concrete implementation. (One can philosophize endlessly - better for dinner conversation than course material.) On the other hand I’ve been enjoying learning from other program participants and trying to develop my own research plan on days like this with lots of break time.
  • Richard’s talk was less relevant and helpful. I think we’re all struggling a bit to juggle 40hrs/week in your classroom with the demands of our lives outside: I wish the schedule were divided into core/mandatory parts which build cumulatively on each other and optional/elective parts which don’t. That way, if we need to miss part (to take a job interview call or push for a deadline or similar), we could plan to miss an optional part and know we weren’t doing much damage to our experience of the month.
AI Alignment: The Field Fri 1 May, Leon Lang 12 responses · energy 3.8/5
Content 4.2 of 5
Teaching and format 4.2 of 5
Difficulty match 2.6 of 5 n = 11
Energy 3.8 of 5
Read what they wrote32
Worked well12
  • I liked the big overview of the field, it's rare to have that. Also the structure with the projects in the afternoon was good.
  • I enjoyed the opportunity to present and be responsible for my thoughts and work, it wasn't quite as much so with any earlier task, save for Stephan's math exercises maybe. It was the case that even if I may have done a bad job at it, I found the process valuable.
  • I like the Greenblatt article, although I had read it before
  • 1. Q&A of project idea
  • Prompting people to think about project ideas was useful, as it motivates more active kind of thinking (as opposed to passively taking in information).
  • The idea of coming up with research proposals and presenting them was a good idea. I also like the exercise of writing down next steps that are feasible and that we will actually act upon.
  • Yeah I mean, career options were lovely, weren't they?
  • The project brainstorming was a useful exercise, and it would perhaps be useful to have more versions of this spread throughout the course (or more warning/prep time)
  • Informative, I like the ideation
  • I ended up speaking (usefully) to [participant] for much of the afternoon, so not much comment - the morning was good, in both the reading and the session on organisation layout
  • I liked the common brainstorming session where people presented their project ideas and Fernando + the other participants came up with comments & questions.
  • I just think it was super useful actually
What they took away12
  • The overview of the field.
  • The clustering, visions and ethics of potential programs, fellowships and employers was new and useful. I would have appreciated more detail, especially high in confidence, about them, but it was helpful as it was.
  • Differences between different grants
  • 1. best way to prepare for coding tests 2. breakdown of funding priority 3. overview of all jobs & training landscapes, very useful.
  • The project idea I thought about I suppose, although it could use a lot more refinement.
  • David's overview of the various different jobs/fellowships/organisations was extremely useful.
  • aisafety.com
  • All of the job org conversations (esp XORLabs, Principia, Stormglass, which I wasn't very aware of) was very useful
  • All the different oppurtunities available
  • Org layout
  • That the Astra Fellowship is "as good as" MATS (according to David).
  • Hearing everyones ideas is just super great
Would improve8
  • I would let the conversation about the overview of the field and its institutions go on for longer. It's nice to get insight on what people who have experience in them share their knowledge, I learned a lot from it.
  • I would still appreciate if we received more concrete opinions, pieces of knowledge, directives that help narrow down the field and set of places to apply. (This covers my above take as well.) And probably this can only be effective beyond a point if each participant (or a handful) gets bespoke advice, which may take too long or need more TA's but I think it's worthwhile. (A counterargument could be that we had access to that during the entire course, since the TA's were friendly and available for discussion, but still making the process more structured helps.)
  • I think Iliad strawmans control pretty regularly
  • The overview of organizations at the start could have been compiled into a document / slide(s), which would have made it a bit clearer and easier to access as a reference.
  • I think the discussing about this plan A, B, C, D got stuck in a not very interesting/useful way and it would have been good to move on from that point to other discussion points before running out of time.
  • Uhhhh instead of drawing the career advice on the whiteboard, why not write/edit it in a Discord message?
  • More time for the brainstorming would have been useful, or maybe some small group workshopping before the big discussion could have also been useful
  • Maybe a more structured presentation of the overview of the funding and jobs landscape

Reading the bars

Every bar is the answers themselves, laid on one axis with the middle of the scale at the centre line. Left of centre is worse, right of centre better, and the widths are comparable from row to row.

1 2 3 4 5, best

A row says n only where fewer people answered it than answered the day, such as a question about readings on a day that set none.