Intensive Teaching · September 2026

The most important things

Before: Join the Iliad Discord and check you can see #intensive-september. That is where you post any material you need to share with participants. If you can't see it, tell Leonard (@leonardbereska on Discord).
During: At 17:40, make sure everyone fills out the feedback form.
After: Fill out the end-of-day log: rate the day and tell us anything that went wrong.
Fill out Who stood out: name anyone who made a positive impression.

Questions about pay, travel, logistics or materials: see the FAQ.

57 project proposals are open for review. Stars, a sentence, or both. Read them.

Your days

[E.1]Debate Mon 28 Sep, leading 26 responses · energy 3.3/5

Overall -0.04 as expected n = 24

Content 0.00 as expected n = 24

Clarity -0.21 as expected n = 24

Motivation +0.08 as expected n = 24

Exercises -0.43 below expected n = 21

Readings 0.00 as expected n = 23

Other activities -0.11 as expected n = 19

Pace -0.04 about right n = 24

Difficulty +0.08 about right n = 24

Novelty +1.21 mostly new n = 24

Read what they wrote36

Worked well8

The game in the beginning -- incredible warm-up, where we were looking for mistakes / errors in simple proofs. This is how most lectures should be starting honestly. Nice balance between exercises, lectures, and papers. Going through exercises together! This was the first time it was done. Delegating it to going through the solutions perhaps also works, but definitely not as good. One of the best days I had indeed -- very applicable math, which finally goes into computational and memory complexity from applicable perspective -- our agents no longer possess infinite compute and memory. This is a great start. As if all my grudges from previous days (where fields had unrealistic assumptions) were finally addressed. And overall the day was very practical, grounded, varied, and at the same time fundamental. Loved it from beginning to end. Lecture on modeling beliefs via graphs -- good, very good, very empirical and theoretical at the same time, Econophysics vibes. Also, Stephan is great lecturer, excellent in fact. Or maybe I am biased to the topic, but really, I was engaged more than most of days.
Speaker was good.
The reading and discussing was separate but also intertwined well.
Reading by our own
Nice choice of reading!
I think this is a really interesting field and one I hope to revisit outside of the Intensive hours - I can definitely see myself working on this further, and I might try to write about it for my project proposal.
Complexity-theoretic framing of debate and interactive verification.
Reading papers and discussing results as a group was quite helpful.kept things very concrete in the second part of the day . Talking about what the curent state of research is.

Would improve18

I would perhaps explain first the idea of applying debate to AI oversight (namely the last presentation). I also found the latter parts of the lecture (related to statistical applications of debate) more interesting than the deterministic debates.
We really need a paper-reading workshop. We had like 30-40 mins per papers, and that time flies. Otherwise can't point out much, I liked the day.
Give more context for the problem at hand.
Some of the reading was not very high quality.
I think it would've been helpful to have more directed readings. Given the limited time available, I'd prefer you offer some hints as to what parts of each paper we should focus on, what information we should try to get out of the paper, or why the paper is important or surprising. The exercises were fine but somewhat redundant with what had been explained in lecture. I'm not sure if it's worth changing them though, interesting problems are fun but not essential for learning.
The narrative is a bit unstructured and hard to follow, and it seems the lecturer wasn't aware of the material that already was taught during the course. It's not a significant problem, but as the lecture is on the last week, contextualizing it better is definitely possible. Some results from AIXI, abstractions etc. are directly applicable to debate.
I prefer slides, but this is something personal
Good lecture day, thanks. I would suggest much clearer planning: for example, trying to adhere to the schedule, stating time allocated for reading. Also, more time for reading was needed, in my opinion. It felt a bit wasteful, after a break, to recall everyone in the main lecture area, waiting for them to be seated, only to announce we had to go back to our seats and read.
I think the content was generally good. The exercises at the top of the day didn't seem very related. I found the content a little demanding today, so in the afternoon, I preferred to read through the notes/papers myself rather than listening to the lectures, where I was starting to get a little lost.
I think doing exercises together on the screen is not super useful, since the amount of time people need to think through them can vary a lot, depending on background. Introducing the AI case a little bit earlier, even just a few sentences during the introduction could be nice as well.
My main criticism of today has to do with (what I felt to be) the lack of lecture time: the lecture parts could have been longer and more formal (but still comprehensive); the little formal content was in the reading/exercises, which for me is not preferable in comparison to the average modus operandi of the previous weeks. I'm not being exhaustive about the things I liked, but I definitely have to emphasise how the way papers were discussed (all together and with slides) is a huge improvement compared to previous days.
The presentation is a bit too math/cs, I think that currently applied debate is different and doesn't use all that machinery. So it may be restructured, e.g. going with practicalities first, and then give theoretical underpinnings. Also, using the latest papers might not be the best idea because a lot of them are slop and better look for some battle-tested blogposts even (so the reading is not well chosen).
I would improve the choice of papers for the first session. They were rather bad papers and were not worth my time.
More motivation at the start, some time to do the exercises.
More time on prover-estimator debate variants.
I found the notes on the complexity classes a bit strange. They try to be introductory but at the same time have to make compromises and present more advanced topics, making it overall a bit confusing. I think having slightly longer notes for the sake of complteness would be very nice.
I don't see how the experimental results connect to the theory except that there's some justification provided by the theory to do debate.
The first part of the day could benefit from having slides. I found that the discussion about Turing machines was a bit confusing. The exercises and notes would benefit from clearer language and a few reminders about complexity theory .

Changed my mind6

That Debate might be extremely effective. I recalled that it incentivizes each side to pull the most extreme arguments, which definitely helps. Also, modeling beliefs via graphs was mind-blowing -- simple yet realistic and effective. We need more theories like this.
n/a
Slightly more optimistic about usefulness of debate
learned about AISI
Debate offers a much more credible path to scalable oversight than pure distillation.
I found the contents very appealing for research.

Other thoughts4

I will have a few interviews tomorrow and will thus likely go to Honeywell to take them, but it will be 1h between 3pm and 4pm, thus shouldn't be too distracting.
n/a
I had a terrible day for personal reasons and it was very hard for me to pay attention today. The feedback reflects on my experience but it is by no means objective.
Super clear exposition of AISI safety cases.

[E.2]Steganography and Backdoors Tue 29 Sep, leading 23 responses · energy 3.4/5

Overall +0.05 as expected n = 21

Content +0.29 better than expected n = 21

Clarity -0.10 as expected n = 21

Motivation +0.43 better than expected n = 21

Exercises -0.05 as expected n = 21

Readings -0.10 as expected n = 20

Other activities +0.52 better than expected n = 21

Pace +0.24 about right n = 21

Difficulty +0.19 about right n = 21

Novelty +1.10 mostly new n = 21

Read what they wrote29

Worked well10

The speaker was clear.
The game was super fun, the reading were very relevant and the exercises very illuminating (ie. the relu exercise). Excellent format and content!!
The game -- great warmup and the way to taste the subject with your own brain. This helps immensely. The exercises -- very physically coded, so I am biased.
Game was fun.
The steg game was great :)
The game
Reading then discussing
I think the structure of the day was good and I liked Stephan's lecturing.
Information-theoretic bounds on steganography H(M) <= H(K) and Merlin-Arthur verification.
The game was quite good.

Would improve12

Present more motivation for the problem at hand; e.g. begin with a cursory overview of cryptography, and how this differs.
This was an incredibly interesting day. There seems to be a lot of empirical stuff which would benefit from more theoretical contributions (ie. a la Shafi). I would really split this into two days.
(Same as yesterday:) My main criticism of today has to do with (what I felt to be) the lack of lecture time: the lecture parts could have been longer and more formal (but still comprehensive); the little formal content was in the reading/exercises, which for me is not preferable in comparison to the average modus operandi of the previous weeks. I'm not being exhaustive about the things I liked, but I definitely have to emphasise how the way papers were discussed (all together and with slides) is a huge improvement compared to previous days.
The game -- it was fun, but lacked structure and general advice. We overthought a lot as well. Also, more work on explanations would be nice.
Similar comments to yesterday on the approach to readings. I liked today's exercises more than some previous ones. I thought the steganography was relatively straightforward, whereas the backdooring was more interesting/surprising to me. So maybe I would have liked reallocating time to go deeper into the latter.
Maybe do two rounds for the steganography game and give more time, or smaller teams? We had a bit of trouble to get organised. Should there be a pedagogical lesson from playing the game? Some truth that should be revealed from playing? Maybe say how dangerous doing research in this area is and how to mitigate the risks?
Today it was a bit exhausting. Some of the slides where heavy in formulas and it was difficult to retain the meaning of the symbols in between slides. I think the content was great, i particularly liked that it was more concrete that what we have done other days of the intensive.
number and define equations with all their quantities on the exercise sheet , better to have more time for some reading and in group discussion than more and everything rushed. I would prefer more of this day and less debate.
So many different presentations and topics, so it was difficult to follow the whole day.
I think the material is good, and I enjoyed the lecturing, but maybe the structure? It sometimes felt more difficult to grasp some concepts than I think it should have, and I found it difficult to work out why, but I think it was mostly the distribution of time per idea - too much for some ideas and too little for others.
Hands-on implementation of undetectable backdoor constructions.
Everyone's a bit tired so I think this day could've been a bit lighter on lectures and more discussion, reading focused.

Changed my mind5

n/a
Yes!! I definitely want to work more on this kind of workk.
It made me more concerned about steganography.
discovered some labs that take students without a master, that work on safety
Paraphrasing defenses are far more fragile against steganography than assumed.

Other thoughts2

n/a
Fascinating module, really sharp math.

[D.4]Agent Foundations · Wed 30 Sep Wed 30 Sep, seconding 27 responses · energy 3.5/5

Overall -0.20 as expected n = 25

Content -0.12 as expected n = 25

Clarity -0.04 as expected n = 25

Motivation -0.36 below expected n = 25

Exercises -0.25 below expected n = 8

Readings +0.17 as expected n = 24

Other activities -0.05 as expected n = 20

Pace -0.28 too slow n = 25

Difficulty -0.08 about right n = 25

Novelty +0.92 mostly new n = 25

Read what they wrote35

Worked well8

Ashe is very enthusiastic and thoughtful, and engaging. That helped a lot.
I think I had high expectations which weren't fully realised, but the day was still good overall. I particularly liked the 3-person discussion.
I liked the philosophical discussion at the beginning of the day
The readings and the free flowing discussions.
Intuition for updateless decision theory and reflection.
I read about open-source game theory, which was interesting.
The speaker was clear
i liked the reading about logical induction

Would improve20

I liked this approach very much- I appreciate that it has tried to link it back to the content covered during other days.
Maybe it should be more broad and more impartial as in less lesswrong-coded, there are a few research directions that fit "agent foundations" idea well, but don't have a corresponding lesswrong post. As Iliad reaches out to people from academia and other backgrounds, it would be nice to accommodate them as well. Know thy audience.
Maybe a bit more of an overview of agendas would be nice, some broader classification perhaps, with blindspots. As the day seemed to be mostly driven by ourselves and readings. Also, the proof in the end pertaining to Intrapersonal game theory took much more time than was available to it, that needs to be fixed.
I see the value in this day, but it's hard to engage with these topics for three days (built upon earlier agent foundations that were similarly philosophical and qualitative). In particular, it is unsatisfying (as part of a 1-month intense programme) to have this many days at that level of intensity (after all, philosophical discussions are tiring) but not have some measurable progress. On the other hand, there were topics like steganography and data attribution that needed more than one or half a day to get a good handle of the mathematics upon which these fields are built. I would have liked to get a better feel for such topics, while contemplating the broader context of the work they entail in the background (as is common when carrying out any scientific research).
The come-up-with-a-project period was fun, but maybe not optimally structured for getting the most out of it. I spent a lot of time skimming various papers and aimlessly musing, didn't get very deep into anything.
Nothing, it was a great day. Perhaps, a more thorough introduction to the various research agendas would have helped.
As usual, I did not have enough time for the reading parts.
there was so much freedom. it would have been great to have a more directed teaching and program. The topics where sparse
It felt a bit hand wavey at times
I thought the day could've had more structure. The readings were very good but I didn't quite tie together for me.
Maybe would be better to bound more the agenda activity. The goal of the activity EAS not very concrete. Having discussions with other participants was great. It was less intensive than other days.
More time on logical induction trade-offs.
It felt like there was very little teaching today. I didn't find the overview at the top of the day to be very useful. The readings were fine, but reading can be done anywhere at any time.
This day could have had more time, especially given that we had three days for agent foundations, which was far more philosophical rather than quantitative.
In my opinion today needed more teaching and less reading / project development time. Most of us are seeing this stuff for the first time, so I think it's ambitious to expect us to be able to develop theories in new and interesting ways - we first need to learn and understand the material better.
I did not find the final talk was very pedagogical. It needs to be modified to make more pedagogical sense. It was too narrow. The "make your own agenda" is hard with the time we had. Agent foundations days should be back to back. It's the only way to make the "make you own agenda" activity make sense.
I found it too philosophical and not formal enough
It seems the day was little empty and lacked taught material or exercises. Given that the first day on Agent foundations was mostly theoretical, it'd be nice to derive some properties and maybe even see some models / dynamics in computational scenarios that can't be derived with pen and paper. However, maybe accounting for the fact most people are tired is also a good strategy.
actuall following in schedule more strictly and bridging the gap between very general statements and concrete theorems that we can propose and prove about agent foundations - in agent foundations day one we already went through readings with fairly high level of abstraction, so i think more concreteness could help.
I was a bit happy about "make your own agenda" as an exercise. The reason is probably that I am unhappy about the pre-paradigmatic state of AI alignment, where there's no established research directions (with established results and conjectures, senior and junior researchers, and open questions), but instead where everyone cooks his own soup and there's little feedback mechanisms and a lot of re-inventing the wheel. Maybe the exercise could have talked about this? This might also not be super specific to today, even though today's exercise triggered it.

Changed my mind5

Decision theories are much more fundamental to safety than expected.
It's interesting that MAISI is working on open-source game theory.
n/a
I updated my perspective on Open-source game theory, and I'm increasingly convinced it's one of the most important research directions (or maybe some of its extensions or modifications) to pursue. It won't solve alignment, but will definitely help in international agreements, illustrative examples and building the foundations for the agentic economy working at superhuman speed (the current one is kinda broken)
time is an important dimension to think about because not all timescales are equal for all agents

Other thoughts2

Superb continuation of agent foundations.
n/a

The day

10:00 Lecture starts

12:30 Lunch at LISA

13:30 Afternoon session

17:40 Feedback form

18:00 Day ends. Dinner at LISA

When	Where
Mon 7 Sep	LISA, 25 Holywell Row, London EC2A 4XE
Tue 8 – Mon 14 Sep	23 Curtain Road, London EC2A 3LT
From Tue 15 Sep	69 Wilson Street, Moorgate, London EC2A 2BB

Resources

Elsewhere