By cohort

November 2026

November 2026 October 2026 September 2026 August 2026 June 2026 April 2026

Rated against what each person expected. 0 is as expected.

Intro to ML Engineering and Transformers Mon 2 Nov, Dan Wilhelm 65 responses · energy 4.0/5

Overall +0.08 as expected n = 63

Content +0.30 better than expected n = 63

Clarity +0.06 as expected n = 63

Motivation +0.33 better than expected n = 63

Exercises -0.35 below expected n = 62

Readings +0.14 as expected n = 22

Other activities +0.50 better than expected n = 34

Pace -0.02 about right n = 63

Difficulty -0.21 about right n = 63

Novelty -0.41 mostly review n = 63

Read what they wrote99

Worked well29

Very clean explanation of the contents, and he asks roughly all the question with a interesting answer.
The fact that some of the prerequisites about transformers were covered again! The reinforcement was helpful
The mathematical background was clear
The one-to-ones And I really liked the second part of the lecture and the presentation of the background with some actual code and papers.
It's really all quite new to me, and I only started preparing about a week ago. But I got really interested and where I didn't fully understand, I still got the feeling this is in reach for me to get on top of.
The lecturer was great, energised, knowledgable and friendly.
The lecture material was relevant and easy to follow, well-curated.
The initial discussions to get to know the other participants were particularly effective. Also, the slide presentation was good, simple enough to be understood by everyone in the audience but interesting.
Icebreakers
I think the step-by-step exercises to get accustomed to Claude Code were really well designed. I had not used it before, and now I have a much clearer idea of what agentic coding entails.
The explanation is super clear
I thought the instructor had a great energy! I also thought having time to meet the other participants in the morning was very helpful and inspiring
The lectures were great, given the neccessary intro.
I felt that my answer for the previous question is pretty applicable: So far it's been super in depth and as someone who feels like I haven't had as strong a foundation in machine learning theory, despite having taken courses in it back at Duke, I found it great as both a refresher and a strength builder for my foundational skills in the field.
I really like the exercises. It forced me to use Claude Code in VS Code. I was using VS Code but for simpler type of numerics.
The broad strokes of what attention essentially is trying to accomplish in contrast to MLPs was quite clarifying. Also, I gained a lot from understanding that W_out for MLP determines fixed directions in the residual stream helped.
The initial "get to know each others" activity worked quite well to start discussions with people The speaker was well prepared and available to answer questions
The networking was definitely very productive. The course was good recap on transformers, and the exercises helped with getting some of the software in place.
Working through transformer lens examples live was particularly helpful
the teacher was very clear and well spoken
The "forced" introductions worked a lot better than I would have anticipated.
Good atmosphere. Everyone was very welcoming!
The exercises are well-organised to remove almost all friction to get our hand on it (e.g., the API keys provided, the well-structured code and folders, the use of uv as package manager).
I liked the Collab snipets shown during the class.
The discussions in the morning
The networking activities were fun and useful.
Watching a transformer being live-coded with the theory worked well for me.
I really liked the exercises at the end.
Speaker was very clear, accurate and engaging.

Would improve50

Maybe don't spent so much time on implementation details like se spoke for 20 mins about softmax today which was excessive. Separate mathematical core of the algorithm and implementation/engineering details while explaining attention etc. make it clear that all of this is a bit alchemy like -- there were repeated questions about theory/guarantees etc. for which there is no mathematical foundation as far as i'm aware and getting it out there early and not wasting time rehashing with later questions would be good
I wish I had more time to work through the Claude Code exercises, especially to try and replicate a paper or even simply to understand better how the attention mechanism work. I think that too much time was spent on ML basics in the morning. I understand that not everyone has the same background so covering having a refresher of the basics might be useful. Perhaps it would save time if that material was assigned as a pre-intensive material: setup exercises or basics for one to cover so that on day one we can immediately start to work more deeply on a single topic or sets of exercises.
- during transitions, the people trying to get attention should be louder. They sometimes spoke too quietly or were reluctant to strike the gong enough to make it loud, causing slow transitions - hard to see code, slides, etc. on the screen from more than ~40% back in the room - would have appreciated more time for the exercises, which contain a lot of practical stuff that it is harder for mathematicians to learn rather than theory that is relatively easy for them to learn. I would have gotten more out of the exercises with more time to do them. - when you have a lot of notation, slides result in working memory overload because you can fit so little on them compared to a board - It seemed like the speaker did not have a good model of the audience. A lot of explanations were in the zone "it's too much to take in if you haven't seen it before, but has little new information if you have", which is a bad place to be.
I felt like there were a lot of questions that disrupted the speaker. Consider having question time less often. Maybe a chairperson could manage how many questions get asked during question time.
It was great to know each other. I appreciate there was official time for that.
I think the claude set up was painful, and I didn't like the exercises.
I thought the intro to transformers could have been alot better. luckily I studied the subject before hand, but these are really technical things and it's really difficult to get it across in simple slides. Actually the 3blue1brown videos were probably better than this. I suppose you have to start somewhere. I was VERY jetlagged as well and nearly falling asleep
we're such a large group, it's v hard for one instructor to effectively "teach" the whole group. consider breaking people up into groups of some number (maybe 10?) and have them work together in that group on some exercise or task, and the instructor can float around the groups helping them individually. maybe also an exercise sheet for people to do indivually or in groups? in short, i feel the sessions would greatly benefit from being more interactive.
I think the screen wasn't particularly visible from various locations, but outside of that it was clear the instructor did their best to convey the general vibe
Maybe some clearer system/guidance for audience questions (when to ask/how, microphone access (though I know the sound guy was trying his best)). Maybe also a bit more brief overview ahead of time of what is the motivation behind/objective for the hands-on activity.
The exercise session felt a bit disconnected. I would have liked to spent a bit more time on understanding the implementations of attention in code.
Maybe a bit more places to sit to listen the lecture and see the slides.
I was close enough to see the display, but even in the second row, if I'd been sitting behind someone tall it would have been tough. I prepared as much as I could, but sometimes struggled with the detail. Though I think I still got quite a lot out of it. I put below that I followed parts of it. But that was perhaps quite good considering my background. This is not about the content, but in the breaks or at lunch the noise level is quite load and it can be hard to have a conversation - in that it's not too easy to hear what another person is saying.
Technical terms should be introduced with definitions/explanations - e.g. MLP should be explained when first referred to, not (much, much) later in the day after multiple references to the acronym. I suppose I would prefer to have the separate building blocks explained then fitted together, rather than being shown the high level spec first and then having its components explained.
Main problem was poor visibility of the slides when not sat near the front. The suggestions regarding a higher TV and another in the middle of the room should sort that out.
The presentation and teaching could benefit from a clearer structure. Perhaps discuss less topics at greater depths. Give an overview in the beginning before diving in. If I was not familiar with the topics presented I would have had a hard time following. Also, the temperature in the room was a bit too low (for my liking).
The coding exercises seem completely unrelated to the lecture. It would be useful to add context/motivation for such exercises. It would also be useful to show us how to perform multi-agent tasks with claude code.
Put dimensions of matrices on the slides and be more explicit with these calculations as its very hard to know what is actually going on. It was not clear that in general one actually works with a smaller dimension in each transformer head, which justifies why one should use V and W_O matricies, whose product is low rank. For me, watching the 30 minute video on the arena website gave a lot more information than this and was more clear.
The lecture was good, although a little bit slow. I would improve the visibility of the screen.
The main improvements I would suggest are logistical. It is very hard to read the screen or hear the voices, even from the middle of the room. There was also confusion about how to get Claude Code to work, which took half of the exercise time to resolve. This is unfortunate because that was the most exciting part of the day for me.
I think it's pretty good actually, but I would say we can have more breaks (I wanted to goto bath room a bit =w=, no breaks) and interactive discussions, because I believe the students here actually have a lot of insights to say and answer each others' questions. For the adam thing, I think it's not just a momentum strategy, it actually performs a bit of normalization (think about a quadratic form loss with diagonal matrix). Btw, a lot of my (newly met) friends report today as proceeding so fast (which is fine, because we would always need some preliminaries and this is supposed to be intensive). But I'm a bit worried that we are going to have it running so fast every day owo(whichi is probably true! I'm kind of ready for that) Thanks very much
Nothing
I think it was a little crowded and difficult to find a space with the number of participants! Other than that, it was good!
I hope we could be seated in the extended monitor area when teaching, so we could refer to the monitor when needed.
Found it strange that the exercises were not connected to the lecture topic at all.
The air got quite thick toward the end - is it possible to open the windows during the breaks? I do not see the reason of the excercises if we just ask the Claude to do essentially all of it. Doing it in these groups (or essentially on owr own) did not see much helpful. But the lectures today were great!
I think it could be clearer when more questions are accepted and when we need to move on in the interest of time.
So far it's been super in depth and as someone who feels like I haven't had as strong a foundation in machine learning theory, despite having taken courses in it back at Duke, I found it great as both a refresher and a strength builder for my foundational skills in the field.
I would try to get a bigger screen and a bigger space to fit us all (which also means you have been extremely successful with this call). We were rather tight.
- The size of the group might be a bit to big for the amount of seating available during the lectures.
Describe the motivation of the technique occurring at first and then the principles about the techniques. Every technology or skills occurred because of some shortcomings from former ones.
I feel that given the background does not strictly assume a lot of ML knowledge, day one perhaps should have assumed a little less understanding of jargon and familiarity of concepts from ML. Perhaps a more intuitive initial introduction, or a very clear additional board with the breakdown of jargon that stays 'fixed' throughout the day, might be quite helpful.
I thought it was a great intro - certain aspects were understandably rushed due to the volume of questions, I think if anything I would've appreciated the slides describing MLP and attention sublayers to include the dimensions of the weight matrices, as it would've been easier to parse what feeds into what, etc, particularly with attention. The steg exercises also were a bit orthogonal to much of the content of the slides, although the later exercises replicating the refusal mediated by a single direction paper were much more connected, maybe that could come before the steg exercises?
Name tags for participants would be great, especially for the first conversations but also in the next few days
Despite many questions being interesting, I feel like they took too much time from the speaker. I think crowd participation could be less disruptive while still be encouraged
The coding part seemed a bit orthogonal to the course itself; however it was very interesting. I would say it was very optimistic, as some of us just managed to get through the first 2 exercises out of the five given, so getting to replicating paper figures still seems unattainable in the time given. Perhaps it's because of the overhead of installing all the required software.
Perhaps the explanations of attention could’ve been broken down a bit better, as mathematicians we are more predisposed to thinking of geometric objects and I think some concrete examples of what query and keys do geometrically could’ve been useful (ie Nandas modular arithmetic toy example). I think, personally, I found the order of topics strange. Only dissecting MLPs after discussing the residual stream, even though MLPs are kind of like the fundamental “unit” in ML.
we're so numerous, the number of chairs or screens weren't at scale :D
Perhaps making people join on the subjects they were interested in was kind of pointless since nobody knew what the subject was about.
Overall, the pace was quite good. Even without a lot of background, I was able to follow many parts of the lecture. Only towards the end did it feel a bit rushed, and it would have been good to have a bit more time.
I think the lectures were great; I would have appreciated at the end/beggining of the transformers introduction, a "big diagram" showing the full structure of a transformer, then going over each part in detail and then returning back to the whole picture!
We are too cramped in the space before the screen. If we broadcast the presentation (maybe just the screen, not the sound), then some of us at the back of the room can follow it on the monitor. On the exercises today: quite cool as concept, but I think it can be improved to teach us (or just explain) more on the techniques (the why and why not).
There are big discrepancies between the schedule and the actual time and sequence of events during the day. It would be better to synchronise them. Also, I think today's introductions are too basic for almost everyone.
The class was good! However, the exercises were somewhat unrelated to the content. I understand that we need to learn Claude Code, but maybe we can focus on using it to develop a Transformer or something.
I think when introducing the transformer and other aspects of AI, it's super important to be clear what parts of the architecture and training and so on is principled (i.e. backed up by good theory) vs what is chosen to maximize efficiency in use of hardware (so in particular these decisions don't necessarily make sense from a theory perspective) vs what "just seems to work" and is still mysterious, and so on. I personally think one half-day is not enough to introduce the transformer. It should be considered prerequisite. I think I would have felt deeply lost if I hadn't previously spent at least a week reading and toying with them before arriving.
The Intro to ML engineering on the google doc looked more interesting to me than what we did today.
I would appreciate more detailed slides with all the technical info on transformers. Also, I think the experiments didn't help in understanding the theoretical part, but they were about experimenting with LLMs. The exercises were interesting, but I would have liked more exercises that reinforce what was explained. Also, the exercises were too long if done as intended within the time window given.
I think the coding setup takes some time, and can be very frustrating. It would be nice if there was a way to initialise the setup without the accompanying frustration.
I really liked the transformers lectures but they did not feel super well organised. Given the audience, I would have preferred a more mathematical treatment, and some of the steps of the transformer were not clear to me by the end of the lecture.
It'd be great if you could use a black/purple pen on the board, i couldn't read red because of the reflection. I would have liked a little bit more engagement during the exercises, though I got through fine.

Changed my mind14

I think now I have a better understanding of how transformers works, even better than before, as the explanations were very intuitive.
I really did like the mathematical framework behind various algorithms.
I (somewhat) learned what a transformer is!
Not much
I had a great talk with the instructor, which made me realise how a lot of the mech interp research is pretty shaky, with a lot of contradictions. This made me interested in looking at popular papers from an adversarial point of view. A next project I have in mind is for example look at the "Refusal in Language Models Is Mediated by a Single Direction" and try to find prompts where refusal does not happen in the direction predicted by the paper.
too early to tell
Not really - just an intro day
I improved my understanding of transformers today.
The one thing I had paid less thought to relative to the entire process in the transformer was the tokeniser, Byte-Pair encoding was brand new to me (possibly because it wasn't covered in the 3Blue1brown series).
I am more motivated to understand these topics, especially in the context of alignment. Working back-and-forth with Claude makes coding more interactive (but it is hard to keep track of) so this is definitely something I want to get better at.
Thinking about the fundamentals again helped me further develop intuition for things. I had not explored using transformer lens and today’s session has inspired me to continue exploring it.
This is the first day with general info on transformers, not so much to change your mind on.
Transformers seem simpler than I imagined
N/A

Other thoughts6

I am filling this out a week later so my opinions are in retrospect and perhaps tainted by that. I actually think one of the following would be better: 1. A better technical introduction to what an LLM / a transformer is 2. The day 2 material about Hugging Face and so on
no
It's a bit weird that the discord is called "Mandate of Heaven". I don't really understand why the religious component comes in there
N/A
The idea of the discussion groups at the beginning of the day was very good, but it was very short. I want more of it!
I am just logging this comment here, in case some else faces this issue with VS code on Windows. Although I had VS Code installed, I had to spend a lot of time trying to get uv to run exercise 0 in vs code. Turns out the problem was that my setup uses anaconda, and what I needed to do was setup "uv". Hopefuly, this is now fixed and should pose no futher issues.

AI Alignment Introduction Tue 3 Nov, George Robinson 66 responses · energy 4.0/5

Overall +0.50 better than expected n = 64

Content +0.44 better than expected n = 64

Clarity +0.64 better than expected n = 64

Motivation +0.69 better than expected n = 64

Exercises +0.22 as expected n = 27

Readings +0.41 better than expected n = 64

Other activities +0.19 as expected n = 57

Pace -0.14 about right n = 64

Difficulty -0.44 too easy n = 64

Novelty -0.11 an even mix n = 64

Read what they wrote81

Worked well22

I think George was a great lecturer! I really enjoyed all of the topics discussed
Discussions are wonderful.
I thought George did a great job today, breaking up the time into organised windows, mixing a bit of lecturing, self-reading, and group discussion. That was well-thought-out and organised and works given the space constraints and large number of participants. George spoke clearly and engaged with all questions earnestly. Also having planned x minute breaks was AMAZING! that should be part of the standard day, there should be an actual coffee break as part of the schedule template for all days; or some down time to catch up on emails or other stuff
I liked discussions in small groups and then opening it up to everyone at the end of each session.
George was a great teacher: clear, engaging presentation, and informative answers to questions. I think it's quite an achievement to get a room full of mathematicians so invested in content that contained essentially no maths.
The motivation for allignment was very well explained, so it motivates a lot the lecture and why it matters.
Great teacher!
I think discussion groups are good overall, especially for cases when materials are relatively introductory. We can go more into specialised topics of interest in the discussions.
A lot of emphasis on active learning.
the sessions format
The small and big-group discussions.
I do like breaking up the hour into lectures/discussions
The structure of the course was a good idea. The Intro->reading->discussion part.
I really liked the format with regular breaks and discussions, made it easier to stay engaged!
Just the instructor's ability to make us engage with the material.
The switching between intro, reading and discussion was very helpful imo.
Discussion with other peers was very interesting.
Alternating between lecture, reading and discussion, in a 1 hour span of time, was particularly effective to quickly digest new topics.
Very energising and helpful lecturer
The session pattern was great. Specifically, skim reading papers and blog posts, then discussing clearly abides by the quote "the whole is greater than the sum of its parts". Then, the frequent breaks helped in digesting the information and refreshing for the next part.
I think the discussion periods (the small “breakout” discussions before we went to wider discussion) were particularly nice, as it got us to hear other interpretations more deeply and to hear about other texts people had chosen
The structure of the sessions: intro talk, reading, discussion in groups, question+answer time, break.

Would improve42

N/A
It would be cool to have a bit more technical and/or hands-on content, maybe some more discussion of partial progress on alignment
Cannot come up with anything specific.
empirical practice of alignment in the companies
(a) a lot of the readings also have a youtube video, some with animations (e.g. huggingface) ... consider, to mix up learning styles, replacing some of the readings with the group watching some of a video explainer? because otherwise it's quite reading heavy ... and like, how much of the 91 page METR report are folks realistically gonna get through in 30 minutes ...
Thought it was amazing, although it is a shame that some discussions were missed. Maybe it would be worth having 5 sessions instead of 6 to ensure we don't miss any of these discussions that I found helpful.
Sometimes the speaker forgot to repeat the question, but I think they became better during the day
Maybe less sessions, the topics were a little repetitive imo. The suggested prompts for discussion was not very interesting imo, I'd rather more philosophical questions.
Today was great!
Increase time in discussion groups.
Some of the readings were too long to finish within the allotted time. This was not a major issue, as not all of the material was essential for the subsequent discussions. However, it would have been helpful to indicate beforehand which sections of the longer readings were more important to focus on.
All seem good to me.
The speaker was rambling a bit some times. I have hard time reading in the room with other people talking. But in general this way of teaching works pretty well.
It feels that the discussions are a bit too detached from the lecture. We mainly have a pattern of lecturer goes over content with some Q and A (these are the best parts), and we do some reading, and then we discuss in small groups. I think it would be much better if we actually went through a paper together (all of us) with the lecturer guiding it or what they find interesting about it with Q and As and discussion built into that instead of separating it. Currently it feels like you could just pre record the lecturer videos (besides the questions asked), read the paper by yourself and talk with the other attendees instead.
The format was good.
Ran out of chicken at lunch
Maybe it would be better to have little lectures where the material is condensed instead of trying to read the original source and running out of time/not getting very far. 6 times the read/discuss block was also a bit much
I found the amount of material and the breadth a bit overwhelming. I don't know how to distil all this down to obtain more narrow, well-defined problems, or how to connect it to the underlying mathematics of AI - though maybe that should be a future goal. I also couldn't see the logical structure of the topics and I think the volume of suggested reading, where I could only ever read a bit of it, added to the overwhelm. Not that there weren't lots of interesting things contained though, really interesting.
I particularly enjoyed today, but I think at least for me I didn't have enough reading time to absorb the readings appropiately!
I would want a more detailed elaboration on each of the materials taught. I would also have wanted a brief explanation of SOTA for tackling each of these problems in alignment.
Too much reading
We did not have time to actually do the suggested readings. It's a bit silly to have us read for 20 minutes when the readings take longer than that. I would have preferred to have the readings beforehand. Also, I want more technical information/discussion. I can get this information (and indeed already knew about it) from popular and well-known resources for laypeople.
This was a good high-speed coverage of the main points of the alignment problem. People around me often said they hadn't finished readings.
There were some issues with lunch. When the second group went for lunch, there wasn't enough food left.
Nothing comes to mind, today wasgreat!
While the material was very well curated, I found it a bit distracting that we had to choose 1 out of X papers to read per topic, whereas most group discussions only focused on one of the papers, so if you chose the wrong paper to read you couldn't actively participate in the discussion. I was also hoping for more technical material, it was too philosophical for me.
maybe reduce the number of sessions that felt too exhausting, and make an effort in formalizing topics. I know alignment is not a well defined science, but still, reading a lot of the posts we were required to read I got the feeling it was too many opinions and less of "this could be a definition, or this paper attempts to formalize something". No hard feelings, I still enjoyed it overall.
It could be more clear what the mathematical aspects are of alignment. Another thing is that, for me, the time to study the texts was rather short.
The readings felt a bit like the blind guiding the blind. I don't really know if it's supposed to be that way because the field is new or because more guidance would have been nice for this part. We were asked to write question on the board, but I didn't really understand what made a good research question to follow up on with these readings? Maybe being more explicit about the learning goal of the reading could have given some guidance for the discussions, but perhaps exploring these ideas as a group was the goal?
Today was awesome. The instructor motivated the material beautifully and guided the interactions very well.
Make it a bit more concrete
I liked it overall, very interesting topic. It was also good to point out the problem of the lack of formal definitions, especially when talking about the higher-level concepts. One should point out even more that this is typical for a developing field. I have the feeling some expectations are "here is the formalism we have built and here is open questions".
N/A
overall really great, not a lot of critisism. At one point questions were being writtten on the board then we came to them last though which meant that it diditn make sense to write them there as we discussed things direct in person first
Not much personal engagement from teacher, unsure how much to expect since there's only been 2 days. Readings were sometimes too long (40-50 page paper iirc) or a bit too short. I ended up reading the thing that was more realistic to digest in the time given. I would have also preferred if there were fewer options, I think 2 would've made it more balanced.
The introduction to AI Alignment was a bit too general, and many of the concepts were not discussed. For instance, one reading was about goal-oriented AI models without properly defining what goal-oriented meant. I was expecting to devote more time to discussing specific concepts to improve my technical expertise, but it was more like a general discussion about alignment and incidents. I guess I was expecting something more high-level or precise. Also, there was quite a lot of reading today.
The lecture was very good, I do not have particular suggestions at the moment.
Clearer desired output after each reading session. Eg: write one summary with 3 bullet points, one open question, one prediction of yours (was asked for on the slides but not discussed).
Can't think of much to improve; the session was very interesting, the readings varied, and the interactive discussions really enabled a lot of nice ideas / worrisome concerns. One thing I was thinking would be interesting to check would be also ask which parts of the reading material people went for (maybe as a poll); some interesting data could come out of people's preferences.
It is possible to add some technical chapters appropriately, rather than most of the content being comprehension and conceptual.
It was great to have discussions and the fact that we needed to actually read articles helped a lot.
The sessions were great today; a good range of excerpts to choose from for each session. In particular, I think we could’ve had more time to read in some of the sections as some consisted of quite large texts. I also think, before the reading, some key goals should be concretely laid out to be looking for in the reading (this could simply be showing us the discussion prompts while we are reading so we can have an idea about what to form an opinion on.

Changed my mind13

N/A
Discussions in smaller groups were good to invite pushback on my priors and assumptions
Now I am thinking about AI safety in a more formal and mathematical way.
I was curious to see that many people here are actually indeed quite new to many AI safety concepts but are very receptive of them.
Not so much, mostly just confirmed what I knew. But I liked the post on think training stories. I think we need something like that.
I had some interesting discussions over dinner about J-lenses and J-space topics.
I am convinced AI is a bigger risk than I thought before. Glad to be here!
If anything, I am rather surprised by how short on time we are as far as alignment is concerned with the rapid speed of progress. AI 2027 was eye-opening.
Yes, I am reconsidering my own definition of goal -directedness and started thinking about subgoals.
That recent empirical studies (toy models of the past ~4 years), even though deep and formal in their reasoning, are mostly built on 'shaky' ground over undefined or ill-defined concepts like alignment, goals and more.
The last two sections were quite harrowing. I don't have an existing background in Alignment specifically, and even though I am working in ML, I realized I was perhaps a bit too dismissive about the rate at which LLMs are progressing. Most strongly I felt the heavy silence in the room as we were reading AI 2027, even after George told us we can start discussing. It felt like bonding, even if it's based on worrying over the same thing. I must say, it also felt like the discussions really opened up afterwards.
I thought the theory about personas (PSM) was very interesting, and offered a new insight into ways of thinking about interpretability that I hadn’t thought about before. Whilst I am still learning, this felt like a very strong lead wrt research direction in the field.
It's very clear why this topic must be researched and what are the possible consequences if we do not solve align

Other thoughts4

I think it would be great to emphasise open problems or research agendas/questions more concretely when giving lectures. Many people here are in position to start doing frontier research relatively quickly and yet are new to AI safety, so giving them some initial food for thought in terms of what can be their research topics, even if very tentative and preliminary, could be useful.
I do think there are too many people in this intensive and the venue is not suited for this. I wonder if it would be better to do the lectures at a lecture hall on a university campus. It is annoying to feel so crowded all day. We didn't even have enough food to feed everyone for lunch, and the lack of space to eat lunch creates awkward social situations. As one of only 4 other women in the program, the amount of crowding, the inability to actually see the board, the fact that you have to somewhat "fight" for a place to even sit every day and a place to have lunch are unnecessary distractions. The actual people are friendly and welcoming but the situation causes anxiety.
Thank you for this opportunity and the organisation!
N/a

Reinforcement Learning and Misgeneralization Wed 4 Nov, Fernando Rosas 55 responses · energy 4.1/5

Overall +0.69 better than expected n = 54

Content +0.94 better than expected n = 54

Clarity +0.52 better than expected n = 54

Motivation +0.56 better than expected n = 54

Exercises +0.11 as expected n = 54

Readings +0.57 better than expected n = 21

Other activities +0.28 better than expected n = 18

Pace -0.20 about right n = 54

Difficulty +0.11 about right n = 54

Novelty +0.91 mostly new n = 54

Read what they wrote65

Worked well18

The lectures were really engaging, and included useful background and comments which helped me orient myself and get a feel for where everything sits. There were lots and lots of questions slowed down the lecture, but this worked perfectly. It was just a function of everyone being so interested and wanting to understand fully the material. Also, it was fun to sit around together and work on the problem sheets during the breaks. Just really productive in every way. I wrote the material was 'mostly new' to me above, but in some sense it was all new, just that, because Fernando posted the slides on Discord the day before, I had a good look at the first lecture yesterday evening.
The topic was about theoretical framework for treating the various aspects about agents (embedded agents, non-embedded, etc.) Many things that we discussed where topics I have given very little thought previously -- much less than they deserve! The session helped me to appreciate *many* questions related various configurations of agents that I was overlooking previously. Ashe is an excellent communicator, and the way, and subtlety, with which he was approaching our questions is something I can learn from.
Very good lecture notes and slides and exercises. Especially the notes are useful and enjoyable to read. I think notes with good condensed information are the best way to learn
I liked the mathematical detail provided by the lectures. In particular the fact that more complexity was introduced gradually, with clear explanation of the motivations for doing that.
The exercises
I just think it was a really, really interesting topic. This is more what I came here for. The first two days were much more "soft" topics which I think most of us have already been exposed to at this point. I was really really happy to learn a completely new, relevant topic in a way that I wouldn't necessarily be able to learn myself. I was also really grateful to get more clarification on what RL really is. It clarified a lot of the discussions and debates that I have heard among machine learning experts.
The lecture was absolute cinema. I understood everything and now I am very motivated about RL. Very well explained the content. Also the open question in the worksheet were interesting.
The teacher is great. He knows how to organise materials in a logical sense and how to present them to give as much as intuition as possible.
Very good content!
Excellent lecturer! I feel like this was the absolute perfect level for teaching mathematicians with no RL experience.
The lecturer was friendly and welcomed all questions
Instructor really emphasized big picture vs details and told us where this was opinionated vs standard. Very helpful for organizing knowledge.
Everything had very well defined mathematical background. So had a concrete goal.
The instructor answered questions very well, and rather patiently.
The material was on a topic about which I had solid intuition and background knowledge. That helped to participate actively by asking clarifying questions and participating in discussions. As a consequence, I understood aspects that previously were not clear to me and which, I believe, will stick to memory for a long period of time.
Good structure! Good division between lecture time and time for exercises.
As I mentioned, plenty of audience questions meant a lot of the ambiguity was clarified, even before I had the chance to realise.
The second set of exercises was amazing.

Would improve33

It was amazing, I learned so much!
I enjoyed the session very much. One thing I would like to see different is more worked out examples that capture the subtle aspect of the abstract concepts and help to understand the motivation behind introducing them better.
Too many questions are being asked, spending a lot of time on peoples personal ideas/thoughts/issues and interupting the flow of the lecture. Especially when people ask questions which will be answered in 2 mins in the lecture anyway. I think one should limit questions during lectures and postpone them until after to direct discussion with the speaker I would've also liked to understand how RL is actually done in practice. I feel like we've seen the theoretical background but I doubt that actual post training for LLM's fits in this and I would like to know how it is done there.
I wish there was a separate day where we could just work through some of the exercises, either individually or in small groups, with the aid of the lecturer when needed. It would also be great to see how to apply the theory to some concrete examples or open problems.
The only thing I would improve here would be to have time to look over the material before the lecture. It's pretty technical and there is a lot of it.
Would have been nice to have some programming exercises.
Today we had lectures that lasted several hours without any breaks (most studies show that people can only meaningfully focus around 1 hour). It made it very hard to follow
I loved the content, finally bringing the rather empirical work we read up to today with some robust formalism on RL. The way the session was conducted, however, could benefit from a couple improvements. Specifically, the two things I wanted to point out were (1) not having consistent breaks (as we did on Tuesday) was very tiresome, especially given how heavy the material was and (2) I feel like the speaker was a bit too permissive on the amount of questions asked, which prolonged the session by probably almost 2h overall (throughout the whole day). Having discussions is a nice thing, but in excess, it felt like it discontinued the overall flow.
Maybe I'd like more examples (posibly computational) or some exercises in the worksheet involving some interesting particular case.
Very little, the lecture was great.
It's good that there is plenty of engagement with the speaker and there are often good questions, but some participants are interrupting the speaker with questions and inappropriate times, or making unnecessary comments. This slows down the speaker, and I find it interrupts the learning for me. Could an announcement be made to at least put up your hand and wait for the speaker to be ready?
My only criticism is that I think that we spent a little too much time stuck on answering participant questions, so the lectures ended up running more than twice as long as expected. Some of the questions are more comments or observations and could probably be gently encouraged to be kept to a minimum.
In the opening of the session I'd give an overview of 1) what is going to be taught, what the new concepts are, 2) what the prerequisites are and 3) why it matters.
I think that starting with the formal beliefs/preference framework before RL is not the most pedagogical. Introducing RL first could motivate a more axiomatic approach later.
Lunch
May be a bit motivation and quick recap of a dictionary of basic stats to Linear algebra. But not much everything else was fine.
It was excellent. I would just include some short break in between
The lecturer and the lecture material were excellent. However, one particular participant, Arman Darbinyan, interrupted every single slide with 3-5 overly pedantic questions, frequently interrupting without even raising his hand. Overall his questions were about minor/irrelevant notation issues which disrupted the flow and was very counterproductive. Each lecture ran overtime by 1 hour because of this. I suggest to leave questions to the end of the presentation, or to maybe allow for questions every X slides, otherwise the rest of the program will be impacted by this behaviour. Regarding exercises, most of them were focused on deriving or proving certain mathematical properties, which might seems targeted to mathematicians. I am an engineer, so I would have benefitted more from a practical exercise, like implementing value or policy iteration for a toy example.
The lectures were pretty good for the most part, but they were delivered in 1.5-2hr blocks which made it hard to assimilate.
The morning lecture formalization seemed a bit dubious to me, coming from physics. I would have preferred a more intuitive explanation on what was really going on. But the afternoon part was very clear and I liked it.
It would have helped to have one or two concrete examples while going through the material
Overall, I think the class today was very good.
lectures were too long and dense; consider having breaks in between, like after 30 minutes; perhaps incorporate discussions into the teaching. otherwise it's very hard to sit there sitting still for 90 minutes concentrating on a single lecture. in terms of content, i feel the mathematical model of RL could have been better motivated; also, fewer topics but deeper
The session was very informative and helped me to forge my knowledge and intuition about the topic. Most importantly, it helped me to connect the knowledge that I had from math to the AI safety context. What I would like to see different: There was a problem session, which was a very fun part, helping to play with the concepts and understand them deeper. However, some of the definitions for the problems were different from what we had during the lecture (i.e. which were on the slides), which led to unnecessary confusion. Some of the slides contained not consistent definitions and math notations as well. Personally, I found the presence of those flaws quite helpful too, as it allowed me to reflect deeper and challenge my understanding before accepting them. However, I believe not every audience could appreciate it. Another thing that I would like to see being done: a brief discussion of the solutions of the questions or providing the solutions as a hand out after the session. Overall, it was a very enjoyable and helpful session!
Nothing! Great session!
The lectures today were great and generally really well presented (I think this sentiment was echoed by the extensive questions from the audience). One improvement is that especially with the second half, perhaps there was a natural breaking point (nothing long, maybe just ~5mins) to fully digest some of the ideas before going onto a separate topic.
There were a few points that were unclear: for instance, pi moved from being a probabilistic policy to a deterministic policy. Some interesting technical points were omitted, e.g. turning pi from a probabilistic to a deterministic policy was suppressed. Ime the absence of such things is typically more confusing, so it may have been better to just speak of deterministic policies. When the ordering of lotteries was introduced, it was initially unclear whether an order was being introduced with certain properties, we were outlining properties that such an order should have, or otherwise. I think the slideshow would've benefitted from far more signposting, making preamble clear e.g. "consider an order, we'll talk about desirable quantities. let mu be a measure such that" and so on. I would have overall preferred more examples. The speakers answers were comprehensive, however, but some of the questions should have been addressed by the slides. Quite a lot of time was dedicated to the exercises, but they were not discussed. I think they were worth doing however It felt like sometimes we got in theoretical rabbit holes and the big picture was a bit blurry, but that may just be me
More of a comment on the interaction with the audience than anything else, but there was a lot of time spent in tangential discussion that was pushed by audience members I thought could have been better spent! Some single slides were discussed for about ~30 minutes.
I thought the lecturer was fantastic and the material was great. I did however think that the lectures went on too long. This meant that we had much less time to work on the exercises, which I think would have been the best way for us to learn, and it is also difficult to maintain concentration for 2 interrupted hours. I think the cause of these long lectures is not the fault of the lecturer but just because there are probably far more questions that were anticipated. Perhaps this is due to the cohort size and lecturers using the same notes as for previous, smaller cohorts? This was also a problem yesterday. It would be great if then maybe taking questions was done slightly differently and/or there were more breaks and/or their is less material covered.
I imagine most of the audience is familiar with vNM-type frameworks. Maybe better to spend less time on this, and more on the less familiar stuff in the second half.
I thought that the plan of the lecturer was excellent. However, I think that allowing unlimited questions and interruptions slightly derailed the plan. I felt that I would have gained more, if the allocated time for lecture and exercises had been enforced more strictly. This is more an issue with some of the students deciding to ask many questions during the lecture time rather than afterwards, but I still think that the lecturer could enforce a little more rigidity and perhaps after a certain amount of time is lost, to ask students to leave all remaining questions until after the lecture is over.
The discussions during the lectures were a bit too long. It is good to clarify a few things at the moment, but a 10 min discussion at the moment we have not yet defined almost any structure... one looses the track.
Would be nice to have more circular areas / sofas for discussion instead of the array of desks. I don't think external.monitors are necessary

Changed my mind9

It helped me to appreciate much deeper the role of abstract models in providing frameworks for studying agentic foundations and look at possible challenges from a unified angle. It opened up many directions in which I can study the topic deeper and motivated me to spend my upcoming weekend to read more on the subject. It also helped me to think more about how to define world models as a mathematician, and how to consider the agent related questions in a theoretical frameworks. This is something I want to continue doing further.
About utility functions. The axiomatic approach really let's you understand what goes behind utility functions. Fernando orally suggested reading "About the independence axiom" which I found very interesting.
Mostly I just understood more what RL is about and what I understood a little better what the different perspectives where.
Agents can be defined (somewhat) mathematically, and now I think I can think on philosophical questions about them.
There's interesting problems in RL for mathematicians to work on!
I thought RL was simply a recipe that you can implement and post-train a model in a few lines of code. It turns out it's a "bit" more than this!
This was perhaps mathematically the most fun it has been so far.
I thought deeper on the underlying learning mechanisms of RL models and on how rewards and utilities can be modeled, which helped me to think more about boundaries of safe and reliable applicability of the current RL approaches. Most importantly, I feel more motivated to dig dipper and learn more!!
Reinforcement learning seems to have a somewhat tractable way to talk about objects that might be relevant to alignment (referring explicitly to model “preferences”)

Other thoughts5

I want to thank Ashe for his time and effort. It was a very enjoyable session, and I learned more not only about agentic foundations but also about how to communicate more effectively in similar settings.
no
After the session, I feel more motivated to dig dipper and learn more!!
N/a
I would really appreciate it if there was a 'moderator' that would notice if the lecture becomes derailed, and would intervene on behalf of the lecturer if that happens. I think that a significant part of whether the day is very beneficial or not, arises from whether or not the lecturer's original plan and time allocation comes to fruition. Also in day 1 (though for different reasons) there was insufficient time for the exercise portion, and that makes a big difference on the learning process.

Agent Foundations Thu 5 Nov, Ashe Vazquez Nuñez 53 responses · energy 3.7/5

Overall +0.04 as expected n = 49

Content -0.16 as expected n = 49

Clarity -0.22 as expected n = 49

Motivation -0.12 as expected n = 49

Exercises -0.25 below expected n = 16

Readings -0.22 as expected n = 46

Other activities +0.02 as expected n = 41

Pace -0.27 too slow n = 49

Difficulty +0.20 about right n = 49

Novelty +1.10 mostly new n = 49

Read what they wrote69

Worked well12

I liked the slides, a short text bc the important is the discussion itself.
Ashe was fantastic in, giving answers to very vague questions. He is a very talented teacher.
The discussions, and the breaking up of the day into smaller chunks
Discussion on the many fuzzy points of this was merited.
I really liked the material's philosophical undertones. It made me think about how the prompts are related to the most results in mech interp. I also had a brief discussion with Ashe, from which I got the idea of mathematically understanding the relationship between prompts and features.
The food was tasty!
I liked the discussions in groups
The begining logical induction was nice, could have spent a bit more time implementing that
Again, Ashe was great. I found the course content difficult though.
Discussion breaks were great. Also, frequent breaks did sufficiently help wrt keeping concentration
The logical induction lecture was great.
The instructor was quite engaging but the side discussion occasionally led the lecture astray.

Would improve48

The discussion is a bit disorganized.
Unfortunately, I think the lectures today lacked structure and coherence. Some parts were interesting, but it was a bit disappointing overall.
I think a whole day on this is way too much. It is really important to maintain credibility, and while this may eventually be recognized as a mainstream science, it is still far.
I would prefer if the empirics were interweaved in the theory rather than appearing at the very end when I'm a bit tired! But otherwise very good day. I think again there should be fewer different options with the readings
The course was very interesting, the glossing over many definitions in the morning felt a bit too quick, but compensated by a steadier rhythm in the afternoon.
more math :D but not everyone would agree but usually the math part is the one that i enjoy the most and also (but it's not only for this prsentation in particular) emphasize more on why it matters for alignment
The philosophical discussions today felt hard to apply to AI safety.
Add a little motivation on each reading topic, I read about AIXI (maybe because I couldn't finish all the reading) and I didn't understand the motivation of this for AI safety exactly.
The discussion drifted a lot, and the slides did not have so much detail, so I ended up thinking about other problems sometimes. But it seems the teacher didn't mind so much whatever we were doing. I don't know if this is good or not.
The morning was relatively useful, but the reading I got was pretty useless and the lecture later derailed into a usless philosphy discussion...
I'd give more structure to the presentations. Share the slides (or just an outline) the evening before on discord/email, motivate the talk and have a clear outline of the day.
More discussions I guess still? But this is a general thing.
The schedule was a bit all over the place and good be a bit simpler. Same with the reading group selection.
The clarity of the presentation.
this day was very philosophical and epistemological, and seemed to lack substance that i could engage with. i did like Ashe's slides, which were not overloaded, and conveyed a single point or two; i also liked how he added breaks at certain times and told us the structure of his teaching. but the explaining of logical induction was hard to follow; i can see though that doing that in a lecture setting like we have is probably quite a challenge.
Bit more maths
I struggled a bit with the vagueness. The maths was a bit fast! But interesting overall.
Material was too philosophical for me, would have enjoyed more technical material. Nonetheless I enjoyed the speaker, very intellectual. Also, some participants interrupted presentations with too many questions again.
I think that there's way too many questions, which kind of made the lecture hard to follow sometimes. I don't blame the speaker, but I genuinely think that more than half the time was spent answering a question. Some of them were insightful, but I'd much rather hear the speaker talk
N/A
Be louder at transitions
The logical induction slides could be made much more clear, for example by giving a description/analogy of the situation, putting in context (e.g. with the Ngo reading provided), and then introducing the math. Also, the speaker should cut participants who ask too many (repeating) questions so that everyone can enjoy the lecture.
A bit too vague. Not really clear goal, what we were doing/wanted to do. I never understood what we were doing/what it was about.
A lot of information to absorb, maybe there is no better strategy here, but some directed questions to discuss I think might have been super helpful. We had some moments for discussion but they were too open ended and could have done with prompt questions, the mental fatigue of previous days is building so that I struggled to formulate some deeper questions myself. Given the intensive nature of the programme I might have prefered slightly less content and more guided questions and exercises to think about, based on the content. It would have been easier to engage with, at a point where brain is already quite saturated with new ideas on day 4. But overall, very interesting. Thanks
Satisfied, only need to add some motivations about some concepts
Would be interested to learn more specifics about the kind of research people are currently doing in this area.
It was very hard to hear the things the teacher was saying, and I would improve the sound quality mostly.
The part about the logical induction criterion was a bit dense since we didn't know all the definition clearly, and it was not entirely clear where the discussion was headed.
It would have helped if the speaker repeated the questions from the audience, and maybe held the mic a bit closer. Some parts felt a bit disorganized too - for example the reading groups part.
I would have preferred a bit more formal background in agent foundation
Nothing! It was super interesting today.
I really liked a lot of the concepts today, but I feel like we could have spent more time looking at practical examples of the concepts
Spending more time on fundamental concrete things before jumping to epmpirical experiments
I have no suggestions for improvement; I thought Ashe was a fantastic lecturer and the structure of the day was great. I did find the course content difficult compared to the previous days, but I attribute that more to my lack of philosophical tendencies than the material being unclear. Although, since I am probably not alone, it may actually be beneficial to give a more structured motivation as to why this philosophical viewpoint of agents is useful for AI safety as I was somewhat confused by this.
Some hand on experiences / practical test / exercises /exercises / practical demonstrations that one could have run/tests in their own laptop would have been helpful
I appreciate somehow the general phylosophical framework upon which the lectures were built, but in all honesty I couldn't follow anything out of pure boredom. I would have preferred a more grounded way of teaching, either fully mathematical (like the RL lecture from yesterday) or in coding. Moreover, in the reading tasks, it seems like "dubious" resources were suggested, I would avoid proposing such lesswrong posts as proper scientific reading (with non peer reviewed arxiv on 50 pages of theory without any empirical validation).
Perhaps the slides could give a little more intuition rather than as many definitions (there was plenty of intuition given vocally, but if one tunes out, it would be useful to also get intuition from the slides).
I do not have any suggestions for today
It would’ve been great if we could have delved a bit more into mathematical formalism wrt agentic formalism. I felt like we perhaps diverged away from this as the lectures progressed which I can also appreciate was an artefact of increased difficulty to define objects.
Overall I liked the lectures but at the same time I think that they did not cover the background frameworks (or the problems the proposed approaches try to solve) enough in detail to engage productively with the main readings.
Can we have access to the slides the day before the lecture? Would be nice to check if I have to review some background before the lecture.
Maybe some toy example in the first lectures could have helped
The mathematics in this lecture were not clear to me. I believe this is, in part, due to the presentation.
No improvements
I have no further comments, I think the class was very interesting today and it was well organized
Clearer explanation (/motivation) of how logical induction relates to AI safety.
I think the class got too dragged due to discussions, and many of the questions were addressed or answered some slides after. I think it would be better to organize the lectures so that for 10-15 minutes we are shown some concrete piece of theory and then discussion, limited to that what was shown, is done later.
I think we went overboard with the philosophical musings today. I would have liked to see more of the "logical induction" paper being discussed at length.

Changed my mind7

The question wheter could exist a widely applicable theory of agency was very insightful, I think is an important philosophical question for the future of AI. I'll think more on it.
I got introduced to some new (for me) ideas in agent foundations which may be somehow tractable. So I think I updated a bit towards the tractability of agents foundation research.
RL and embedded agency. I had never learned of how embedding the agent can break the theory so much. Bayesian inference: I didn't know logical inference and found it very interesting. This was the most thought provoking lecture so far for sure.
I learned about Löb's theorem today, which was surprisingly surprising and made me reconsider if mathematical logic might be cool. I am still pretty uncertain of the applicability to AI safety though.
N/A
Again, some of these things are incredibly hard to define without getting very philosophical. But here is where I see the remit of trying to do this using mathematics
Turning english into rigorous math is hard. i've long known this but it always nice to be reminded of it.

Other thoughts2

I can't be here tomorrow morning, as I have to run a tutorial session, but I will come in in the afternoon, around 2pm most likely.
N/a

Alignment in Practice Fri 6 Nov, Margot Stakenborg 52 responses · energy 4.1/5

Overall +0.63 better than expected n = 49

Content +0.55 better than expected n = 49

Clarity +0.43 better than expected n = 49

Motivation +0.84 better than expected n = 49

Exercises +0.62 better than expected n = 26

Readings +0.64 better than expected n = 47

Other activities +0.51 better than expected n = 45

Pace -0.12 about right n = 49

Difficulty -0.16 about right n = 49

Novelty +0.22 an even mix n = 49

Read what they wrote76

Worked well16

The game
The first lecture was very well-done and I think it's really important.
The lectures in the morning were amazing, really interesting and also well-structured. The challenging activity in the afternoon was so educational, and there was a good amount of time for pre-reading. The fact that the challenge was going to follow also helped increase my focus on the pre-reading, because everything learned was about to be used.
The final game was very insightful and I enjoyed the discussion a lot.
I really enjoyed the red/blue teaming exercise. The "adversarial" setup provided a motivating pressure to think broadly and deeply about how to solve real problems in the present, and I liked this hands-on feeling.
I really enjoyed the red team/blue team exercise. It was a very meaningful way of engaging with alignment and I hope we do more in this style!
The best part of this class was the debate organised at the end. I really liked the format, where we spent 2 hours learning about the topic and then formed teams to create safety challenges and break them. Thinking in this way really cleared things up for me for a lot.
I really enjoyed the exercise! I got more out of the reading because I knew what to focus on and retain for the exercise.
The reading exercise
The blue/red-teaming activity. The whole combination of the day today.
Good lecturing style and structure
It's nice to hear different topics each day. After having some days with fairly philosophical discussions it was nice to have a day with complete focus on practical aspects.
The exercise in the afternoon had a lot of very interesting discussion:)
Very engaging speaker
The solid mathematical background of GR and SGTM provided good sanity checks
Today we covered a material that likely everyone in the AI safety field has to know well, which is the alignment pipeline. Despite that, this material was among those that I regard as my weakness, as it is not using much scientific intuition that I possess and is more about the current state of the alignment approaches. Given that, today's session perhaps was the most helpful one for me in order to feel more prepared for the AI safety field.

Would improve49

-
great lecture.
not much, it was overall good
I didn't love the exercise. For me, it was more like a debate thing rather than designing a system. The only thing we did with my team was put every single thing we read about on a system. Like, its not like we know a lot of variants of ways of training and post-training. I understand that anyway the goal was to discuss and become aware of these subtleties, but it was weird nonetheless.
The group exercise was very interesting, but I would do it in smaller group to have more chance for everybody to participate to the discussion
I think I would prefer more information about the real world and what actual companies/governments/other organizations do that playing a game.
The day was great. I would maybe improve the repartitions into pretraining/post-training/deployment in the afternoon activity
It was great
Good
Everything was alright!
overall very good, Margot spoke clearly and engagingly and had a fun exercise at the end that got everyone interacting well with each other
Amazing. Extremely valuable day. Only possible improvement would have been some advance room-bookings, more break-out spaces in the afternoon when we were doing the team activities.
Maybe try to stop discussion when it becomes too long, to have enough time to visit every remaining concept in the lecture.
I wasn't here in the morning, but the red/blue/judge exercise was great! Maybe even more time for more iterations, maybe a slight increase in competition pressure, also I'm not sure if the intended procedures and scope for red/blue teaming were clear to everyone. But it went well overall.
Nothing, good studying material and good game.
Today was the best day so far! I think we should have given space for people to read on more than one topic, specially for the exercise. the readings were not that long, and it would have been more interesting if people were familiar with all the areas (pre-training, post-training and deployment).
Would have appreciated more technical material, slides were too descriptive, and figures are of low quality/very pixelated, hardly readable. Also seems like we rushed through and skipped many slides, felt very rushed. Again, some participants are asking too many questions to frequently.
First part of this day was great, I think the teacher was excellent and very knowledgeable. Evals/audits are probably my favourite topic. All I'd say is that she spoke and went through the slides far too quickly. I think it would've benefitted from a bit less detail done a bit slower. E.g. more should have been allocated to private reading. The structure of day 2 was great in this respect, I think more days should be like that I think the group exercise drifted a lot. I felt like both groups lost sight of "safety" and the fact that the fine tuning API was the pressure point (it should make people think of emergent misalignment). I think more intervention from the TAs would be great.
this seemed okay
It will be helpful to have name tags
**posting again as I would like to correct my previous, somewhat unfair assessment in my last point 4** 1. This would be a great lecture for opening the week. It was quite general and didn't go deeper in any topic (not necessarily bad). 2. The group exercise was fun and creative, however too generic. The teams would benefit if the rubric was a more specific, realistic scenario where we would focus on pretraining only or agents escaping the lab only (for example). Asking from a team to come up with a solution for *all* problems in training, deployment and alignment in 30 minutes is defocusing. Less is more. 3. I would give clearer practical instructions for the group exercise (what each color means, what each number means, in which room to convene etc) and post them on the board (not discord). 4. I wouldn't spend so much time to entertain all questions from the audience but rather focus on delivering the content. At several points the conversation felt repetive and it was missing the point. As a result, some great ideas - both on the slides and the reading material - were not touched upon at all in the classroom due to lack of time.
I think the blue team vs. red team game was particularly helpful to think about pros and cons of several methods to make models safe. However, I wish I had a better grasp of some of the techniques in order to make my case in a more cogent way. Perhaps, it would help to have more reading time or cover a bunch of these techniques in the lecture in more detail. Also, I think it would help to wrap everything up at the end of the game with a short discussion open to everybody on the main weak points of these techniques and how they could be improved.
It is good today
It was actually a very good class, and there was very little to improve.
I would put this lecture first in the week. It was pretty generic and would fit well as opening. It didn't examine any topic in depth and the group exercise, although fun, it was also too generic to be helpful. This would be more efficient and focused if we were presented with a narrower case, e.g. "there is a legitimate researcher that does such and such and wants to achieve X while a malicious hacker wants to use the model for Y. How do you block one while allowing the other?". Some detailed scenario that will lead the teams into thinking about *a few* particular problems and techniques and not *all* the problems of training, deployment and alignment at once, making the debate unbounded. Side comment for the lecturer: I would leave 'watercooler' conversations for after the class and prioritise delivering content. I felt a great part of the lecture was spent in informal and unhelpful chats.
Overall good. Feel I learnt a lot here.
The day was excellent overall. I think that one thing that would have improved the final stage even more, would have been to have been given a 'model answer' for blue and red team, after we all tried out the debates ourselves. This would have been useful because many of our arguments were of course uninformed or not the most coherent, whereas seeing a well thought out red/blue team discussion ideas from the lecturers/ judges would have made our final understanding more concrete and precise of what is feasible and correct, and what is not.
Nothing, Margot to a fantastic job! She was really organised, knowedgable and designed the exercise well.
Maybe designated space to work in group
It's hard to say what to improve, as the course was very interesting and well-structured. The exercise, whilst engaging, was very unclear to my team (group 2, red team). It felt like we had no clear scope and (at least) I misunderstood a lot of the intermediate tasks, but more because the general purpose was very vague. Perhaps it felt quite rushed as well. All in all, I'm sure this can be a very exciting activity, with minor improvements and clarifications. This being the end of the first week might also have affected it given our low energy levels (not saying energy drinks could compensate).
More discussion
NIL, thoroughly enjoyed it!
maybe the exercise is too long ? not sure though it's interesting
The topic and structure of the blue team/red team was quite vague, but maybe that was on purpose
Great!
The lecture could give a little more details about different alignment techniques (though I understand that we skipped them because the participants asked too many questions).
Probably nothing, today was briliant, especially the redteaming.
Good day over all. Very good lecturer. The TA could've been a bit more organised/focused and stricter with the rules of the exercise.
Today's talk is a bit vague
Maybe enter more into details the mainstream approaches of Alignment techniques and give a taxonomy.
I thought the topic today was super interesting. Maybe I would like to have more time to discuss the topic with the teacher. But anyway, the exercise afterward was also super interesting.
The organization of the afternoon exercise was a bit confusing. I preferred the format the other day where groups were announced on discord, except for the letter to number correspondence part
It would have some more technical details on the tree stages, even though I understand time is limited. In a similar spirit I felt the talk for the debate was very broad, so while it was fun and interesting, I could have gotten more from a somewhat narrower task.
Not much; I really enjoyed the day. I do think that the length of activities was a wee bit long but overall great day!
Nothing
I feel like we skimmed through a lot material and many things were difficult to follow. There was a lot of open-ended discussion but I didn't get much from the slides.
Today was great
Make it clear in the last exercise that while pitching the AI model one should stress on quantifiable proeprties, rather than philosphical debate (I agree this is much needed but it cannot be the base line).
Likely, I would be slower in describing the alignment pipeline. For example, describing in more detail the evolution of the pipeline from the historical perspective would give us a stronger perception of the new types of risks emerged over time with more powerful models and what were the newly introduced measures/criteria/open problems that appeared along the way. All those was touched in the session but more structure in covering it and a bit slower pace would make it more effective.

Changed my mind9

I actually got a favorable impression of the J-lens (yesterday some people were convincing me it's not useful).
I tried out some ideas that didn't wind up working.
Be in the shoes (virtually) of a person taking decision at a company makes me realize how hard the decisions are and that there are a lot of trade-offs there.
It started to feel more reasonable/concrete that (and how) AI companies can create models and give the public access without necessarily causing unbounded catastrophic consequences.
N/A
I think the sessions are getting derailed by open ended discussion during the lectures. I think this is good in general, but it is harder to follow the thread occasionally because of a fair bit of back and forth, some of which should really happen on a 1-to-1
Personas get baked in during pretraining. Deeper understanding of the methods to avoid harmful behaviours from beginning. Way better understanding of the whole training process. Mostly confirmed my views that getting safe AI will be very hard, next to impossible.
I think my understanding/appreciation of just how important a role the model deployer has is to mitigating misuse other than adding harnesses/flagging output. There are plenty of ways to implement safeguards to ensure bad actors do not get access however I understand that a.) this is hard and b.) this is still very much an open problem.
I would say that I didn't change my mind in a big way, but prepared it more for the AI safety field :)

Other thoughts2

the reading chunk was rather long, that could have been shorter, or have more in-person activities
N/a

Debate and Scalable Oversight Mon 9 Nov, Leonardo Costa Lesage 54 responses · energy 3.5/5

Overall -0.31 below expected

Content +0.07 as expected

Clarity -0.30 below expected

Motivation +0.17 as expected

Exercises -0.46 below expected n = 39

Readings -0.09 as expected

Other activities -0.41 below expected n = 29

Pace +0.06 about right

Difficulty +0.35 too hard

Novelty +1.11 mostly new

Read what they wrote82

Worked well13

The basic mechanism and hope with debate came across very clearly.
I thought the readings were really quite well scoped considering the amount of time we have to digest.
The teachers had good energy
Convening in groups to review content we read was very useful, as I had definitely failed to interpret some things that other people had managed to.
The readings were very interesting
The final part was about the practical examples and research outputs demonstrating the advantages of the debates. This was very helpful!
I mostly enjoyed the more mathy stuff (debate, cx-debate, factor graphs), and the second set of readings.
I found the reading about debate vs consulting interesting.
Some sanity checks using tablue during the exercise sessions
Nothing really. The 25 minutes to read a technical paper weren't worthwhile. All the fun math was ignored. Turing machines were defined but then all the complexity classes were given the cold shoulder. This made little sense. I know these things so wasn't bothered but it was clear that not much thought had gone into making this day a worthwhile day.
The reading and presenting sessions
Having two presenters worked well: it gave dynamism to the presentation.
The first discussion was insightful and the activity of finding incorrect proof was fun.

Would improve54

Write out the math on the slides more explicitly (as a mathematician, I'm not afraid of formulas, and think they would automatically clarify a lot of things).
I think the basic theoretical framework could have been presented a bit more clearly. There was quite a large gap between the prerequisites posted on Discord the day before and the introduction to the formalism in the first lecture: the prerequisites introduced basic notions at a very high level, whereas the first lecture began immediately with a highly technical and detailed formalism. Perhaps reading about the Debate setup in advance, or going through a mini-tutorial, would have eased the initial encounter with this new material considerably.
Temperature was low at first but are now okay
NO
I don't have anything in particular to say
The lecture material today was too detached from how debate is actually implemented in practice. In contrast, the reading material was actually very relevant, so if lectures could resemble the reading material it would be more insightful. Ultimately, I was hoping to understand how to apply techniques such as debate in practice, but it was too theoretical from my perspective.
I think the topic is very interesting and relevant, but for me, the explanation was confusing, not very well organized, and too complicated. I think that fewer concepts, but with more time to thoroughly study them, would be better. For me, it would have been more interesting to understand debate than trying to recap all the techniques used in debate (in particular a concrete fellowship project) because it was very hard to follow all of them. For instance, I think it is unrealistic to think that a person who has not studied this topic before can read a research paper on the topic in 20 min and comment on it. This makes the student rely on an LLM, and then the learning is to try to read a summary an agent has produced? I am not an expert, but I do not think it is a very efficient teaching technique.
I hope those slides would be available on discord.
Clearly communicating times (especially restart after lunch) and following through there could be improved.
It may have been interesting to actually implement the "debate" protocol in a game between the Intensive attendees; it felt like a really natural thing to do/try.
The talk became progressively more heuristic (totally understandable) but I felt as if the Prover-Estimator Debate section was a bottleneck for understanding all the downstream alignment applications. I would've spent more time spelling that out, perhaps with examples, as bayesian networks weren't in the prerequisite material.
The reading was way too long for the allocated time. We also had another sessions where the questions got out of hand and obstructed the exposition.
morning I felt needed some clarifications a little better of some of the points, but that being said I am not native to formal comsci theory. Felt like the afternoon was much more followable and interesting personally for me. Overall good though, thanks to both of you.
The beginning of the first lecture just started immediately with the wrong proofs. It would have been nice to have a brief overview of what/why we were going to do today.
I would have liked more details and time spent on the AI safety case that was discussed in the end, it seemed very interesting and close to practice. Also maybe running an experiment in a debate setup would have been instructive. The reading was very interesting!
I think the readings could have been better. The first reading (on whether llms cheat on lean) was a little confusing, and it was not clear how we should interpret the conclusion
I enjoyed the content however I feel the lectures were harder to follow today than in prior days. I mainly believe I felt this way due to my lack of familiarity with the concepts. Perhaps a bit more motivation/toy examples could have been used to get the points across.
A bit dry and abstract. A few more concrete examples would've been nice.
I think a misunderstand between the audience and the presenters is like we're studying these idealized mathematical models then the empirical stuff is different and it's hard for mathematicians to make sense of it. I think adapting the class around this difficulty could help reduce the friction. (For example, in the reinforcement learning lecture learning about Bellman's equations then seeing how it's used in practice was really cool).
Nothing to improve
-- The provided notes were containing many flaws on the level of notations and language. This was slowing down the reading. -- The definition of the main concepts, especially, of the cross-examination debate, could be done slower, with more accentuated and carefully designed examples before and after. A one-to-one map of the abstract definitions with the example would be very helpful. -- Some of the proofs, I think, could be optimized. (In particular, of DEBATE=PSpace) Overall, it was a challenging session for most of the participants. My opinion on this might be biased, as I was familiar with most of the theoretical background and my motivation for this topic, but nevertheless, I can related to the shared opinions about it being challenging. I very much enjoyed the final part, were practical applications of debates and their advantages were discussed.
I think it would have been helpful to have access to all the slides used. The speakers did point to Lens Academy but either they were hard to find or weren't there. As someone sat at the back, I couldn't really see the screen so this would have been really great.
Regarding the presentations, I'd say the lecturers had very good intentions, but they were not properly prepared, which led to them sometimes getting lost in the explanation, having to repeat stuff... I feel it is what would happen to me if I were to speak about something I know about but did not practise too much before that. It would have help to have a layout of what we were going to cover today and the motivation, and maybe a full-day recap at the end. At any given point I kinda knew locally what we were doing but I was more lost as to the global direction. I feel that the connection to AI safety at the end could have been fleshed out more. As a mathematician, though, I am happy that the complexity theoretic proofs were included, due to the comfort of the familiar. As for the reading, maybe too much time was devoted to it, for the benefits we gained from it. Also some of the references seemed a bit old (which I understand in this field is hard to keep up) which made us wonder how much the results were relevant to the state of the art. I enjoyed the second set of readings more than the first (I feel the concepts of prover, verifier, world model did not require the reading to understand). It feels like the times for reading, as well as for the breaks, were taken a bit loosely, and then we ran out of time at the end. Also, at the beginning the lecturers had trouble navigating the large amount of questions they were asked, which kinda broke the flow of the lecture, although I could tell how they got better at managing them as the day progressed, sometimes you have to cut them and move on.
Send slides on discord before lecture. The slides on prover estimator and graphs were a bit harder to follow.
Perhaps add more details about the specific of RLHF and RLHAIF. The some parts of the descriptions felt a bit vague.
too fast on the demos, i guess a lot of attendants aren't too familiar about complexity and the prerequisites were given a bit too late to at least prepare a bit more precisely
I did not particularly like the content. I wish we were doing slightly more practical things, and I am not fully convinced by the importance of formal verification given other much bigger issues in safety nowadays
At some point, I feel like the explanations could have been clearer and more compelling.
there were rather a lot of readings this day
More structured discussion groups, some moderation for them.
Faster pace
in general good
Very cool topic and very good presentations.
The first part about incorrect proofs was too easy, and the rest was too hard, was my impression. I think it needed reminding of definitions and notations and taking a slower pace, maybe more time to read and think on our own would also have helped. Somehow, the presentation was often too vague or too quick for me. I didn't have the slides in advance, and normally I've looked ahead at the slides, which helps me follow during the day. Re the lensacademy, I wasn't sure if I could open it already on Sunday, since it said it was a one-day access. But not all the content was found there either.
The session today was quite heavy, and I'm not entirely sure if this was because I am not used to the topic, or because of how it was explained. I felt like I would lose track of what is happening every other sentence. It is hard to explain what didn't feel right but I think I generally didn't see the bridge between the formal explanations and the illustrative scenarios. I think the focus should have been on trying to make a better intuition and connect it smoothly with the formalism behind it. I can also say that most of the other participants I talked to had similar impressions of the session today. One note, however, is that the quiz clarified a lot of things, so that was definitely a good addition!
It was interesting, but as Leonardo said we got "in the weeds" a bit too much, I feel it could've been a bit more high level and broader. Especially when we got onto debate on graphs. I thought the last slideshow was good. I enjoy evals and so liked the readings
The mathematics could be presented more clearly. The proofs related to complexity classes were hard to follow for me, because it was not clear what was being proved in the first place.
Spend time on some of the foundational exercises which helps understanding the different kinds of debates and the mathematical/ statistical hypothesis behind them.
Slides should perhaps be a little less condensed with information and should be made larger. It can be a little hard to follow especially being a large class and having to sit slightly further away.
I think today was rather poor despite the fact that the content is very fascinating. Both instructors today did not seem very well prepared. In fact none of the ideas were explained well, and this was such a missed opportunity. I have heard about safety via debate from other mathematicians and from the sound of it, this is a very beautiful topic. I was not too thrilled with how it did not seem worth my time. I left around 4:45 simply because I couldn't be convinced to sit through the last hour.
Nothing.
Maybe more reading on our own and discussions part rather than listening. I think with this topic would have been easier to try to read the lecture notes with pen and paper at my own pace. Because the explanations were nice but not for a first time learning/reading about the topic.
The material today went very fast. I think the Lens academy plugin made things very confusing. I think a lot of material has been stuffed into one day. It would have probably been better to skip the paper reading session in the afternoon and just do more exercises. It would have also been nice to see some example of toy decision making problems in each of the frameworks.
The topic was not properly introduced. The lecturers should give a brief 5-min recap of Languages, Decidability, and complexity theory (at least to fix notation / naming conventions).
Too many interruptions for questions, harder to keep a coherent narrative in mind
It was a very good lecture. Nonetheless, I would have perhaps started with the motivation (the very last topic today) instead of the more formal presentation. I had the impression that some assumptions (and the comparison with more realistic systems) were not well articulated.
Exposition quality was somewhat poor, and the speaker did not seem to understand the background of the audience. Speaker seemed under-prepared. It became clear during the break that many people (perhaps most) in the audience (a) do not the definition of Turing machine and (b) are not familiar with common complexity classes. It would have made sense to cover this, and I think there would have been time to do so had the rest of the material been structured more carefully. On the other hand, I am familiar with this material and found the lecture incomprehensible anyhow, while the papers on which it was based (e.g. https://arxiv.org/pdf/2311.14125 and https://arxiv.org/abs/1805.00899 ) clearly and precisely explain the same material in a small amount of space. The talk would have benefited from the speaker just stating the precise definitions of things, instead of trying to hide them constantly. Slides rarely contained enough information to understand anything about what was on the slide, including basics like "what do the symbols stand for".
I would teach background knowledge first. The Delta between the suggested two videos and the point where the lecture started from was unexpectedly large. It felt that only people with a solid background in complexity theory and/or formal logic could follow the session. I was not one of them and had to 'fake' my way through the topics. Therefore, I would either spend an hour or two building this up, explaining all basic concepts from scratch or require students to spend one day at home catching up (but with explicit instructions and material) prior to the teaching day. In the grand scheme of things, I can see the point and usefulness of the whole "debate" debate. I believe the jist of it can be delivered with much less - or even none - of the formal (and not so transparent) maths presented today and be still impactful. The reading sessions were fine.
In general, a good session. For the first reading exercise, the time was too short. For the second reading exercise, the assigned time was better. I had the feeling that the room was colder today than last week.
More accessible readings
This is a very fun topic, it's going a bit fast but I understand we can't slow down due to the content. I spent most of my time on the basic definitions
There seems to be many questions about how unrealistic the most simplified setting is. Because every topic is presenting a point of view towards alignment, it may be useful to present what is the speaker position towards: how realistic/unrealistic the approach is? Having an early disclaimer may help with that.
.
Maybe having more examples of problems to understand better the abstract part of the theory

Changed my mind11

Maybe debate is really a scalable oversight
Debate might be more useful than I expected. Did not know about the safety case skech based on debate. Need to look in more detail.
Not yet!
Debate itself is an interesting technique to potentially ensure aligned behaviour, however I personally would still require some extra convincing of its effectiveness.
Before the session I was feeling motivated to work on debates. Now I feel even more motivated. Despite the critical remarks, the session was very helpful and the speakers have done a great job.
The research area of debate is much more formalized/theoretically developed than I thought.
Imperical research is important but the motivation should be to unveil the theoretical background for it.
Safety via debate is a very nice topic. I need no convincing on the aesthetic front. At the risk of sounding like a broken record, it felt like my time wasn't valued.
That oversight can be scalable
Glad to see some theoretical properties of debate (computational complexity) which formalizes that debate could be more benefitial than a single AI, even on the iterations of this idea like cross examination.
How lean works, I had heard a lot about it but did not know how it works.

Other thoughts4

I was not feeling well today, so I could not really focus on the reading and I skipped the exercises for some other work. Might have to stay at home tomorrow.
N/a
Thank you!
I had to go early today :( im sorry, I read the last paper but I lost the discussion

Solomonoff Induction and AIXI Tue 10 Nov, David Quarel 0 responses

Reading the bars

Every bar is the answers themselves, laid on one axis with the middle of the scale at the centre line. Left of centre is worse, right of centre better, and the widths are comparable from row to row.

much below below as expected above much above

Pace and difficulty are a direction, not a quality: the centre is the target and both ends are a drift away from it.

on target one step off two steps off

A row says n only where fewer people answered it than answered the day, such as a question about readings on a day that set none.