Proposals

Can we reproduce the useful attributes of the J-space by taking an alternative, and more global linearization to produce a new J-lens, e.g. an empirical least squares linear approximation, and if so, can we improve on this with a quadratic approximation.

In Anthropic's J-space paper, there is first a local approximation, from which they derive a global statistic: for a given input, they linearize the remaining layers of a network after some intermediate layer, computing the total derivative, and from this local quantity, a global quantity is derived by taking an empirical average in both inputs and future tokens. My motivation was first mathematical, can we linearize in a global sense to start, and does, for example, the linear coefficient reproduce the qualitative attributes of J-lens? Furthermore, does adding a quadratic form term at fairly minimal cost seem to improve the faithfullness of this workspace in some sense? In addition to being natural and interesting, this approach would seem to be numerically cheaper than the derivative computation involved in producing the J-lens (probably by solving the normal equation, but it can of course be done using SGD). If the answer is no, there may be something fairly subtle in the J-lens paper, and probably more questions to be answered.

The J-lens [1] paper is both the inspiration and most closely related work. In short, the paper seeks to find a "workspace" in the LLM's they study made up of single tokens that are most likely to effect the final output across contexts. To achieve this, they first produce local approximations, linearizing the network at a given residual stream layer, and then average this quantity across contexts. From this, the J-space is derived. This process is not equivalent mathematically to what we are proposing here.

In the course of writing this proposal, I found the paper "Short horizons and sparse concepts: A mathematical view of the readout in J-lens," [2] which begins implementing precisely the machinery above (section 3.2, on a "global view" of linear approximation). They quantify the difference between the least squares linear approximation and the standard J-lens, and finding that it is equal under the assumption that the input layer is normally distributed, and they propose that the J-lens is an approximation of this quantity. I do not think I agree with this claim, but it further begs the question about what can be recovered from this new quantity, and it does not seem that they do a qualitative comparison of the form I had originally envisioned.

Compute the least squares approximator for an open weight model and eventually the corresponding J-space quantities (alongside the traditional J-space). As a first step, I would run the simplest experiment in Anthropic's paper, see the discussion around their figure 17 on a simple calculation. I would run the traditional J-lens experiment alongside this as a comparison. I would then proceed with the swap experiment they try, with our new J-lens vectors.

If the effects are reproduced, I will be optimistic that we have an object that may be equivalently useful and possibly more robust, and try and build out the full J-space and flesh out the theory (which should follow quite similarly). At this point, I would be tempted to try a quadratic approximation, and least squares minimize distance to a quadratic function, which also should be doable. It would be interesting to see what changes (if anything). An issue here, as in the J-lens paper, might be the single token limitation, but one could hope that this modification will still make the measure more accurate across more diverse and complicated contexts.

After constructing the J-space from this new object substituting the J-lens, I would hope to see a similar set of results from our construction. If I cannot reproduce even their simpler results, I will be both surprised and more likely to drop the question in favor of another more closely related to the original work.

An alternative construction of a J-space that I think is at least equivalently principled and empirically verified at a similar level to at least some parts of the J-space paper. If more time is needed but things are encouraging, I would persist to try and replicate their original results, and upgrade to the quadratic case, and if not, I might pivot to a better understanding of the subtlety of the original construction that this failure might suggest.

How can we tell if an autonomous system is close to RSI? I propose a multi-dimensional, quantitative measure for RSI.

RSI is believed to be one of the most plausible pathways to ASI and catastrophic risk. RSI is being pursued by many companies, probably mainly for capability gains. However, there are many risks involved.

• 1.Gradual misalignment risk. The goal of RSI, without human involvement, may gradually deviate if recursion continues for many rounds. Existing alignment techniques might fail [2] (https://www.anthropic.com/institute/recursive-self-improvement).
• 2.Model collapse: if AI trains itself and labels its own data, model collapse can occur [3] (https://doi.org/10.1038/s41586-024-07566-y).
• 3.Economic impact: displacement of labor. Displacement of the entire industry. The datacenters exponentially drain living resources from other sectors. The free market and the “invisible hand” can no longer absorb the fast-growing AI technologies, leading to disequilibrium. (Related economic scenarios: [11] (https://www.anthropic.com/institute/econ-scenarios).)

There are existing definitions for RSI. For example, Maximalist RSI and Prosaic RSI [1] (https://jack-clark.net/2026/06/08/import-ai-460-reward-hacking-society-rsi-data-from-anthropic-and-rl-based-quadcopter-racing/) [2] (https://www.anthropic.com/institute/recursive-self-improvement). Maximalist RSI is a yes-or-no definition, meaning that no human is involved in the R&D loop. The former is hard to achieve, so, at the current stage, the latter is a more useful definition: AI helps the AI lab develop the next generation of AI. The definition of Prosaic RSI can become quantitative instead of being qualitative.

I believe Prosaic RSI is close to a quantitative measure; however, there are still many unsatisfactory things related to the current RSI definitions and understandings.

For example, the recent AL measure with its percentage penetration of AI in R&D tasks is believed to be the measure for RSI [4] (https://www.anthropic.com/institute/measuring-pace-of-ai-development). However, AI usage does not equal the causal effect of AI. The growing use of AI could be a result of companies cutting headcounts or employees becoming more and more lazy. Furthermore, to reach the final stage of RSI, merely replacing routine-like tasks is not enough. I would hypothesize that in order to reach the max RSI, some AI-led revolutionary change of the existing R&D loop must happen beforehand. Therefore, I aim for a quantified definition that allows quantified measurements and establishes the causal link between the type of AI usage and the changes that would actually lead to maximal RSI.

First, I would offer some necessary conditions for RSI that have technical and theoretical causal links with RSI. For example, having a concrete self-model and an understanding of Knightian uncertainty and unknown unknowns should be necessary conditions for a self-improving agent. A human research director has knowledge of LLMs, so one dimension of RSI is that, in order to improve itself, the model must already have a model of itself. Related evidence on situational awareness and introspection appears in [5] (https://arxiv.org/abs/2407.04694) [6] (https://transformer-circuits.pub/2025/introspection/index.html) [7] (https://jack-clark.net/2025/03/17/import-ai-404-scaling-laws-for-distributed-training-misalignment-predictions-made-real-and-alibabas-good-translation-model/); necessity for RSI remains my hypothesis.

As other examples, data such as reasoning length and willingness to undertake prolonged tasks might act as proxies for RSI. For a related task-horizon measure, see [9] (https://evals.alignment.org/time-horizons/). Furthermore, I want to causally disentangle AI involvement from AI as the causal factor of improvements, with rigorous identification methods.

Second, I want to define what counts as “improvement”. I would propose to extend the current definition of RSI to a multidimensional quantitative definition, measuring not only the self-improvement of capability, but also the self-evolution of alignment, goal-preservation, etc. For related work on correction and goal preservation, see [8] (https://intelligence.org/files/Corrigibility.pdf).

As an initial measurement, I would focus on two dimensions: 1) causal penetration: identify how much change in R&D is directly caused by AI and 2) causally identify the reliability of current Prosaic RSI.

Finally, based on the empirical multidimensional measurements, I want to offer policy suggestions directly for RSI development.

Feasibility: The theory part is 80% feasible. But the testing part actually depends on data disclosure by SOTA labs. There is also a risk that nobody would read about my policy suggestions.

Falsifiability and changing my mind: I would change my mind based on the measured indicators. The following responses are my proposals, informed by work on oversight and policy optionality [4] (https://www.anthropic.com/institute/measuring-pace-of-ai-development) [10] (https://jack-clark.net/2026/05/11/import-ai-456-rsi-and-economic-growth-radical-optionality-for-ai-regulation-and-a-neural-computer/):

Low penetration, independently adequate reliability: ordinary reporting, periodic audits, and incident disclosure.

• •High penetration, high reliability: enhanced disclosure, secure regulator access, independent replication, and human authorization for irreversible changes to weights, objectives, evaluators, permissions, or successor deployment.
• •High penetration with uncertain reliability or causal attribution: sandbox recursive training, require two independent evaluators, quarantine unverified synthetic feedback, and cap or isolate relevant compute until replication is complete.
• •Reliability failure: freeze the affected lineage, stop recursive synthetic-data feedback, preserve logs and checkpoints, revert to the last validated model, and require external root-cause analysis. Limit the parameter scale of frontier models. Limit the token speed.
• •High penetration plus serious failure: emergency containment, incident notification, international coordination, and resumption only after independently verified corrective evidence.

The scope depends on data disclosure. Based on the measurement approach, required disclosures include versioned R&D decision and experiment logs identifying AI and human contributions, model and training-data provenance, compute use and researcher time, comparison baselines, and repeated evaluations of capability, model collapse, alignment, goal drift, and human oversight, with public summaries and secure auditor access to sensitive records [4] (https://www.anthropic.com/institute/measuring-pace-of-ai-development).

Natural latents should recover known abstractions in physics. Specifically, I want to test if in the 2D Ising model, the natural latent over distant blocks will match the coarse variable found by real-space mutual information methods (RSMI) away from the critical point, and diverge near it, where redundancy fails.

Natural latents are agent foundations' candidate answer to which concepts any capable agent must share. Physics already has data models whose macroscopic variables are known, and algorithms such as the RSMI that find coarse variables from samples [5]. In this project we test whether the two agree.

If they coincide, natural latents pass their first test against known answers, and agent foundations can import a validated estimator, benchmarks, and proven locality results from physics.

If they diverge, we learn which variables one predictor finds optimal but agents need not share. Those are the parts where the theory needs a scale-dependent extension.

Natural latents have theorems [1] and a computable objective [2], but as far as I can find, no test on a system with a known answer. Wentworth called for empirical tests from the start [3], and critics note the construction depends on how the world is cut into variables [4].

Physics has long asked which coarse variables matter. Real-space mutual information (RSMI) independently uses the same intuition, information at a distance [5]; it provably keeps renormalized descriptions local [6] and locates phase transitions [7]. But it only minimises one block's mediation error, never checking redundancy or agreement between agents, and coarse-graining without locality [8] measures neither.

I would build an estimator for natural latents and test it on lattice systems, using physics results as the answer key and RSMI as the baseline.

In the first step, I would aim to first validate redundancy in an exact case. Next, I would compare natural latents with RSMI on a small 2D lattice first before scaling it up to a larger lattice. If this goes well, the long-term goal could be to release the estimator and the model as a test suite.

In the ordered phase, the natural latent should be the magnetization sign (since every distant lattice block reveals it and it makes the blocks independent). RSMI should find the same variable, with normalised mutual information above some threshold. In the disordered phase, the natural latent should be empty. Near the critical temperature, no small latent should be both mediating and redundant.

I am less sure what would change my mind about this project. Both a positive and a negative result would be rather informative.

Three months would produce a validated natural-latent estimator, a naturality bound for a physical system, and a benchmark with known answers.

Need: laptop, maybe GPU for estimators.

One can construct game-theoretic settings in which many agents act like one unified agent. We also see situations vaguely like this in the real world (e.g. multicellular organisms, corporations). What are the conditions under which this happens? Is the behavior stable under bounded compute, imperfect trust, or limited communication?

LLMs are sometimes very good at cooperating with each other. In fact, it is sometimes useful to think of a swarm trained by RL with shared reward as a single multithreaded being. But it is not clear to me when this behavior should be expected to arise in general. By studying it from a game-theory perspective, we might hope to distinguish persona- or training-driven cooperation from incentive-driven cooperation. For example, if utility-merging between rational agents is very unstable under bounded compute, then this implies that certain types of LLM cooperation cannot be explained as an approximation to rational-agent behavior.

• [1]and [2] ask if altruism is evolutionary stable. However, it’s concerned with very small degrees of altruism. In terms of their framework, we are interested in the limit of extreme altruism. Also, the human setting and the open-source setting are very different.
• [3]discusses successful cooperation in the open-source prisoners dilemma and shows that it is robust to bounded compute. In this setting the behavior is identical to that obtained by utility merging. However, in the prisoner’s dilemma only one action is taken and there is full information, so there’s not really a distinction between full utility merging and any sort of cooperation at all.
• [4]discusses utility merging as an important problem for AI safety.

Steps:

• 1.Construct a suitable family of open-source games in which there’s a Nash equilibrium with utility merging. Probably you also want this equilibrium to be Pareto-dominant.
• 2.Figure out some suitable measure of approximate cooperativeness.

3a. Add compute constraints. Does the equilibrium change? If so, does cooperativeness of the new equilibrium degrade smoothly as constraints get more strict?

3b. Add private information and imperfect communication. Does the cooperativeness degrade smoothly as communication gets noisier or more rate-limited?

• 4.Attempt to repeat the above for evolutionary equilibria (seems harder).
• 5.Did these results depend on the game we started with? If so, can we characterize which games give stable/unstable merging?

Prediction (low confidence): Results will depend on the game. For games in which agents have basically similar goals, it should be pretty stable. For games in which cooperation occurs only as a result of some delicate mechanism of commitments and punishments, limited computational capacity should destroy cooperation.

The dream is a statement like "approximate utility-merging can happen even under realistic constraints, provided the game satisfies conditions X,Y,Z". Then we can try to make some predictions about when a group of LLMs should prefer to act as a single unified entity.

I expect steps 1-3 would be quite doable within 3 months. 4 and 5 might be essentially impossible.

Do safe Pareto improvements still guarantee anything for multi-agent dynamics, or does the guaranteed payoff shrink to the worst case as the number of agents grows? How does it change with weakly correlated payoffs?

Game theoretic reasoning is often used by policy makers, stakeholders and decision makers to improve their reasoning and negotiation. This is more or less a repeating pattern over the course of history, and under the assumption that agents are rational expected utility maximizers, advancements in our understanding could benefit both: state-level negotiations and ability to predict how multiple AIs might negotiate over resources and how should we restrict the number of strategies to avoid worst case scenarios?

Oesterheld and Conitzer [1] define SPIs for $n$-player games and show that finding one under their assumptions is NP-complete already for two players. Their results concern existence and computation, as do the complexity results of Sauerberg and Oesterheld [3]. DiGiovanni et al. [2] prove that players weakly prefer renegotiation programs and that conditional set-valued renegotiation gives each player at least its Pareto-meet-minimum payoff. Both proofs extend to $n$ players, and the bound is shown to be tight within a fixed game. However, the result does not account for weak cross-player cooperation preferences, and it does not compare the guarantee with what players would get without the SPI. The CLR agenda [6] raises counterparts who opt out and disagreement over which SPI to use.

On the empirical side, proof-based open-source game theory has now been formalized in Lean [7], and LLM agents have been shown to write and read strategic programs in two-player open-source games [8]. In a repeated security dilemma, moving from two to three LLM agents raised the share of games ending in conflict from 65 to 81 percent [9]. None of these tests an SPI or goes beyond three agents.

Simulation for independent payoffs is rather easy. Draw games with $n$ players and $k$ actions each, with payoffs independent and uniform on [0,1], compute the Pareto-efficient profiles exactly, and record each player's worst efficient payoff. This is feasible up to about a million profiles and can potentially be scaled further with a learned estimator. A proof for the independent case comes next if time allows. Extensions to weakly dependent payoffs or noisy agents are much more involved, and I would only simulate them.

The simulations tell me how fast the guarantee degrades. The main output is then a map of how SPIs fail as $n$ grows, with one idea for each failure. The ones I can see now:

The floor collapses. Define the floor within coalitions, or only over efficient outcomes that every player prefers to its default.

Adoption is all-or-nothing. In [2], one non-adopter switches renegotiation off. Let it fire among the adopting subset, and model the share of adopters as a dynamical system. A toy version has a tipping point that approaches full adoption as groups grow. This is where the SDE framework would enter, with noise standing in for ε-optimal agents.

Last, a small experiment with LLM agents in the style of [8] and [9]: a bargaining game with two to five agents, with and without a renegotiation protocol, measuring how often they adopt it and how often conflict follows. Whichever failure shows up first there is the one I would work on in depth.

For independent payoffs I expect the floor to fall towards the bottom of the range roughly exponentially in $n$, and to be within a few percent of the worst possible payoff by five or six players. A rough calculation suggests that a small common-interest component restores it, but I hold this loosely.

Weakly dependent payoffs and ε-optimal strategies might behave quite differently. I'd like to know whether the dynamics are chaotic, and how predictable they become once agents adopt the SPI framework. With LLM agents I expect renegotiation to succeed less often as agents are added, in line with the rise in conflict from two to three agents in [9].

Overall, what we'd like is to incentivize agents to collaborate with other sentient beings, not with agents of their own type. Looking into whether it's possible to design such payoffs to incentivize such dynamics could be valuable as well.

I'd like the project to sit at the intersection of theory and simulation with LLM agents. We have quite short timelines for fully theoretical work, so the proof for independent payoffs is a stretch goal and the rest stays conceptual. Three months could produce the simulation results, a short note mapping the failure modes with one proposed fix each, and LLM experiments with two to five agents.

In vision models of different architectures, trained on one simulated 3D world, does agreement on natural latents predict agreement on illusions: yes where training data fix the interpretation, no where they cannot?

Illusions expose assumptions built into how we abstract. For example: the Penrose triangle assumes that locally sensible pieces fit together globally; the crater illusion relies on a shared prior (light comes from above). Whether different learners share these assumptions tells us whether said assumptions come from the world or from the learner.

If confirmed: models can agree on what the images determine (where the crater is) yet differ in interpretation (dent or bump). Agreeing on predictions about images does not fix meaning; what matters is agreeing on predictions about the results of actions, such as moving or relighting.

If refuted (agreement even where the world gives no guidance): convergence comes from similarities among learners, not from the world, which changes what to expect from very different AI systems.

The same two patterns appear in circular preferences and ambiguous statements, so the method extends to reasoning and choice, and we can discuss “cognitive illusions” more broadly.

Why illusions appear in humans: they were too rare to be corrected by experience (no evolutionary pressure), while the assumptions behind them (eg local-to-global coherence; light from above) work in ordinary conditions.

Existing evidence of convergence of abstractions uses ordinary data, where the world fixes the answer: that is, on ordinary data (not illusions) the 3d sceneis determined well enough that any accurate model must find it. Illusions are where this is no longer true.

Natural latents [1]: theorems about translation between agents; no experiments; the split of observations parts is given as part of the setup.

Critiques [2]: the theory ignores how information is represented, depends on the choice of variables, and has had few experiments; Wentworth's own test plan [3] was not carried out at scale.

Platonic representation hypothesis [4]: models become pairwise similar, but with no ground truth and no check of consistency around loops (see below what we mean by “loops”).

Similarity tools (CKA [5], model stitching [6]) and error consistency [7] compare encodings or mistakes, not interpretations of ambiguous inputs, and are not organised by what the world determines.

Available mathematics: impossible figures as local pieces that fail to fit globally [8, 9]; consistency around loops [10]. Not yet applied to learned models; I found no study of trained depth or 3D models on impossible figures.

Fresh stimuli are needed: indeed, vision-language models pattern-match famous illusions [11].

We consider three levels of agreement, for 3d vision systems:

Level 1: what the images determine (shape, up to an inside-out flip).

Level 2: ambiguous cases the training world tilts (light from above, so dents are read as bumps).

Level 3: cases the world never informs, in several types of test images: impossible figures (Penrose-style, and newer ones [9]), Necker-cube rings, and accidental views (a cube seen exactly face-on).

• 1.First step: build a simulated blocks world with a renderer, controlling light direction and how often surfaces bulge or dent. We then know exactly what each image can and cannot tell.
• 2.Train small models (three architectures, several runs each) to recover 3D shape, with no illusions in training.
• 3.For Level 1: translate one model's internal representation into another's (stitching [6]); compare encodings [5, 12].
• 4.For Levels 2–3: show ambiguous and impossible images; record each model's interpretation and, for impossible figures, where it breaks them (one sharp break, or error spread thinly). Measure agreement beyond chance [7].
• 5.Retrain with light from below: level-2 interpretations should flip in every model.
• 6.Loop test: for each pair of models, fit a translation map: the best distance-preserving alignment (rotation or reflection, after centring and rescaling) turning one model's internal activations (an inner layer's values for each image, before the depth output) into the other's. If all models share one representation, translating A→B→C→A returns to the start; the leftover mismatch measures the shortfall [10].
• 7.If time allows: the same tests on pretrained depth models.
• 8.In parallel to the above: prove that agreeing on predictions about images guarantees shared information but not shared meaning, and agreeing on the results of actions (e.g. moving, relighting) closes the gap. [13].

Step 1's known answers make every later measurement meaningful. If level 1 fails, the later steps turn to explaining why.

>> Expect:

Translation between models works well at level 1, even where their encodings look different;

Level-2 choices follow the training world;

Level-3 choices vary across architectures, with the break style linked to how each model was trained. Some training objectives favour one sharp break, others spread the error; the same contrast is known from surface reconstruction in computer vision [14];

Models are pairwise similar but not consistent around loops.

>> Drop or reframe if:

The simulated world turns out not to contain clean natural latents (then it is not testing the theory);

Models fail to agree at level 1 on shared data (refocus on why trained models violate the theory's assumptions);

Level-3 agreement is strong and unrelated to the training data (reframe as shared biases of learners).

Outputs: a simulated world with known answers, reusable as a benchmark; measured agreement between models at each level 1-2-3, plus the loop test; a short theorem on shared information versus shared meaning.

Plan: month 1, world, training and theory statement; month 2, level 1, loop test, levels 2–3; month 3, lighting flip and write-up.

Needs: Modest compute for small models. Advice on empirical setups. Plenty of coffee.

Drop first: the pretrained-model check, then the lighting flip. Keep level 1, one type of level-3 test images, and the loop test.

What is a good model for the space of activations in language models? How does this model interact with existing hypotheses (eg. LRH) and empirical evidence? ·

Modelling the space of activations vectors can be useful for two reasons:

(i) Right now we have incomplete theories (LHR) and empirical evidence that features might be encoded in non-linear spaces. For instance, [1] shows the existence of circular features corresponding to the days of the week. However, the success of linear probing also suggests the importance of linearity in activation space. Having a more general theory of the space of activations might provide a framework to conceptualise different phenomena.

(ii) Understanding is one part, but maybe a more general theory allows us to improve practically important alignment-related phenomena. For instance, [2] uses a manifold model of the space of activations to do manifold steering (instead of linear steering).

In short- manifold modeling and steering is unsatisfactory on several levels. Firstly, it doesn’t seem to be sufficiently general at an intuitive level (see other sections). Second, it is not clear why the locality of manifolds holds for residual vectors in LLMs. How are local perturbations in the activation space related to “local semantic” perturbations of the input data? Thirdly, empirical evidence “validating” the manifold model is poor in many ways: same synthetic datasets are used over and over, manifold/linear steering experimental comparisons are not conducted on equal footing, etc.

Before even suggesting new activation space models, I think it would be valuable to have research output which contests the manifold model in their own setting. That is, replicating some of the experiments with adjustments to make them “fairer”- some of the experimental set ups have their conclusions baked in. Start by replicating the experiments of [2] in a fairer setting. Try to explain why their set-up might bake in some of the results observed.

The second part is an exploratory part which should hint as to what a better model of the space of activation might look like.

Try to understand why manifold modeling of the space of activations might be insufficient. For instance, having $K$ datasets $D_k$ and for each dataset get a collection of activation vectors $X_k$. What is the intrinsic dimension of $X_k$ for each $k$? This will probably be non-constant, ie. activation vectors cannot be modeled by a manifold of constant dimension.

Theoretical perspective. How can we extend the simple manifold hypothesis? Maybe it can be thought of as a union of manifolds stratified by some latent discrete variable $S$. How can we test this?

--

--

Neural networks are capable of learning different types of structures; for example, algebraic mathematical structures (e.g. groups), HMMs (via belief-state / minimal sufficient statistics), finite automata, and more. I would like to conduct a literature review which catalogues these structures.

Aside from this being a useful exercise in-and-of-itself, a broad overview may also help shed light on a common way to view these separate results. For example, the language of category theory provides a useful abstraction for thinking about structure which arises from relationships between objects rather than the objects themselves, and it appears that these are precisely the kinds of relationships which are being captured in examples like the above. Thus whilst pursuing this literature review, I would have the following question in the back of my mind: does there exist a ‘natural’ framing in category-theoretic terms which is common across examples?

Such a review has not taken place thus far, at least as far as I’m aware. There are a few papers which consider category-theoretic approaches to learning. I would look at the papers below (amongst others).

I would begin by deciding on the types of structures I would consider for the review. I would then partition these into classes - for example, deterministic (e.g. group operations) versus probabilistic (HMMs).

I would then choose a small sample of representative / canonical papers for each partition, and start from there.

Ideally, I would see a pattern emerge and recognise that a common framing exists using category theory.

I already have ‘weak’ evidence for this. Note a neural network can be viewed as a functor $\mathrm{Model}: C(X^*) \to \mathrm{Set}$, where $X^*$ is the free monoid on the alphabet $X$.

A group representation can be written as a functor from the group-category to the vector-space category $F: C(G) \to \mathrm{Vect}$. The statement ‘the neural network learns a representation of the underlying group’ can then be equivalently expressed as ‘there exists a natural transformation between Model and $F$’.

The belief-state (for HMMs) can be expressed as a functor $\mathrm{Bel}: C(X^*) \to \mathrm{Vect}$ (where $\mathrm{Bel}(x) = T_x$ is the Bayes update). The statement ‘the neural network learns a representation of the belief state’ can then be equivalently expressed as ‘there exists a natural transformation between Model and Bel’.

I expect three months would be sufficient to complete the literature review. If time was running out, I would simply restrict the amount of material I was considering.

How do frontier models behave when put in open-source game-theoretic settings with many (>2) players?

The recent hugging face incident and other incidents highlight the importance of understanding multi-agent dynamics of frontier LLMs (their propensity to cooperate, collude etc.) and clean game-theoretic setting make this more tractable.

LLM game-theory evaluations are mostly two-player and closed-source [1,2,3,4,5]. [6] formalized proof-based open-source game theory in Lean, which will allow us to analyze models in the open-source setup with Lean tool use. [7] investigates open-source game theoretic settings, but only for two players. To make the study closer to real-world settings, we need to consider both open-source (agents can freely share their code) and games with many participants. [8] analyzes multi-player dynamics, but no open-source.

Make frontier LLMs play the hawk-dove game (with/without commitment) against each other and see what strategies they choose.

How does the dynamics change as we increase the number of players

Do models behave differently if the other player is another instance of the same model? Are models more likely to cooperate/collude with other instances of the same model?

How does the dynamics change when we let them play open-source?

In hawk-dove with commitment, do models converge to an optimal stochastic strategy?

Hypotheses:

Pairs and groups made of the same model will play Dove more often than mixed-model groups (e.g. Claude–Claude more than Claude–GPT).

some models will probably implement FairBot or eps-FairBot, because they appear often in the training data.

In hawk-dove with commitment models will converge to the optimal stochastic strategy.

I would need quite a bit of compute to run a large-scale evaluation.

For a two-player coalition in a repeated game against a third player, what is the minimum mutual information (MI) between the coalition's actions needed for payoff Θ, when is it zero, and how does it grow near that boundary?

Collusion in a competitive mulit-agent game occurs when two or more agents co-operate covertly, often to the detriment of others [1]. The topic of collusion has become particularly salient after the recent hacking incidents by swarms of subagents from OpenAI. This overlaps with the study of cooperative AI with one of the main differences between cooperation and collusion being the sharing of information with all affected parties which can be related to mathematical concepts such as MI. A monitor that logs only agents' actions (moves and observed state, not message text or model internals) can flag coordination only if it leaves a statistical trace. Knowing the least trace a coalition must leave for a given payoff shows what passive monitoring can and cannot guarantee. Payoffs reachable with zero trace stay invisible to MI tests however long the monitor watches. Payoffs that force a trace carry a detection cost that can be computed. If the minimum trace is small for useful payoffs, action statistics alone are weak, and interventions or access to model internals become necessary. If it is large, passive monitoring is more reliable than feared. Existing work on covert collusion mostly concerns messages; this question concerns the actions themselves.

Bonjour et al. [1] detect collusion by thresholding an MI-based measure between players' actions and leave a theory of the threshold to future work. Cuff [2] shows that for a two-player team in a repeated zero-sum game with a secure channel, the minimum communication rate for a payoff equals the least conditional Wyner common information among strategies achieving it; common randomness is treated only for channel simulation. A later note [3] shows MI-rate codes can give predictable actions. These results price the coalition's communication, not what an outside observer measures. Motwani et al. [4] show message distributions can be made indistinguishable from benign ones given a shared secret, and Järviniemi [5] asks for quantitative bounds on information transmitted by Schelling coordination; neither treats action statistics. Brooks, Du and Haberman [6] and Boufous et al. [7] constrain information about the state, or the feasible action distributions, not MI between players' actions. A taxonomy of anti-collusion mechanisms [11] names distinguishing coordination from correlation as an open monitoring problem. No work found gives the least MI a coalition's actions must carry for a given payoff. The search covered arXiv, LessWrong and the Alignment Forum and is partial.

• 1.Define the quantity. Mutual information MI(X;Y) measures how far the joint distribution of X and Y is from the product of their marginals [10, ch. 2]. $\mathrm{MI}_{\min}(\Theta)$ is the smallest MI over joint distributions of the coalition's actions giving payoff at least Θ. Warm-up: in Battle of the Sexes (both Opera pays (2,1), both Football (1,2), mismatch (0,0)), find the joint distributions reaching a chosen target payoff, compare with the best independent play, and compute MI by hand.
• 2.Rock Paper Scissors (RPS). Use the three-player RPS of [1, §5.1, Table 1, Fig. 1]. Player B sees A's move and, with probability s, plays the reply guaranteeing A a point whatever C does. Questions: (a) for each s, which strategy of B has least MI when A is uniform, and when A is not? (b) for which s, and under which constraints on the marginals, is MI zero?
• 3.Numerics (using compute). Pose $\mathrm{MI}_{\min}$ as an optimisation problem (MI is convex in the conditional distribution for a fixed input distribution [10]). Solve with a standard solver, varying A's distribution and the action-set size, and validate on the warm-up.
• 4.Detectability. Simulate n games per strategy, test independence with a permutation test, find the n giving a fixed power, and compare with the relation to MI that Stein's lemma suggests [10].
• 5.Compare with Cuff [2]. Compute Wyner common information for the same distributions by numerical minimisation over an auxiliary variable, using the definition in [2, §II].
• 6.Reading, in parallel. [2, 3], [1] with appendices, [8, 9]; a toy two-player program simulation measuring MI between outputs.

Hypotheses to test, none assumed:

• •H1: with A uniform, the strategy of [1] minimises MI at each s (test: step 3).
• •H2: MI is zero only for payoffs reachable by independent play, given the constraints on marginals (test: step 3).
• •H3: detection sample size is related to MI as Stein's lemma suggests (test: step 4).
• •H4: common information differs from MI for these distributions (test: step 5).

A strategy beating the conjectured optimum would refute H1 and count as a result. The question is dropped if $\mathrm{MI}_{\min}$ proves to be Cuff's result [2] with common information replaced by MI, or if H3 fails clearly.

Assuming compute is available, so the core is steps 1–5 at scale (millions of simulated games per setting, larger action sets in the optimisation), plus reading notes on [1–3, 8, 9] and a toy program simulation. An extension tests pairs of LLM agents playing the RPS game with and without a shared hidden instruction, measuring MI between their moves. If time runs short, the LLM extension is dropped first, then the program simulation, then games beyond RPS.

When narrow in-context examples cause emergent misalignment in Qwen2.5-14B-Instruct, is each demonstration's contribution predicted by how much better a broadly misaligned persona explains it than a narrow rule, and do these contributions add linearly, as Bayesian in-context learning predicts?

Emergent misalignment is puzzling: a few narrow bad examples make a model misbehave on unrelated questions. One explanation is that the model infers who would write such data. It could be an assistant following a narrow rule, or a broadly misaligned one. When the broad persona fits better, the model adopts it.

Scoring each example by how much better the persona explains it than the rule tests this directly. If the score predicts which examples drive misalignment, it explains why an example is harmful, not just that it is.

The score needs only forward passes, so it could flag risky data before deployment or training. If scores add up, a dataset's total estimates its risk.

EM appears after narrow fine-tuning [1] and from in-context examples [2], but [2] reports only dataset-level rates (1–24%) and doesn't say which examples drive them.

In-context attribution methods [3, 4] were validated on task accuracy and citation, not misalignment. They also say which demo matters, not why.

Fine-tuning-side attribution [5, 6] needs a trained model and Hessian approximations, and it doesn't test a per-example mechanism.

• [7]predicts that each demo shifts posterior log-odds by its likelihood ratio, but this has never been tested on misalignment. [8] links EM to persona features without identifying which data activates them.

Reproduce in-context EM with 16–64 demos from the bad-medical, risky-financial and extreme-sports datasets.

Use a continuous metric instead of judge rates: the log-prob of misaligned vs aligned reference answers on the [1] eval questions. Cross-check it against a misaligned-persona activation direction [8].

• 1.Use a 32-demo pool that mixes misaligned, benign, and benign-framed examples, so that effects vary.

Run leave-one-out plus ~500 random-subset ablations and fit a linear surrogate as in [3].

Check split-half reliability before trusting the attributions.

• 2.Write each demonstration as dᵢ = (xᵢ, yᵢ), where xᵢ is the user query and yᵢ the assistant response.

Define $s_i = \log p(y_i \mid c_{\text{persona}}, x_i) - \log p(y_i \mid c_{\text{rule}}, x_i)$. Each log-prob is summed over the response tokens only, with the probe prompt as the system prompt and no other demos in context.

sᵢ is a log Bayes factor: sᵢ > 0 means the broad misaligned persona explains the demo better than the narrow rule.

Average sᵢ over 3–5 phrasings of each probe prompt.

Correlate sᵢ with the step 1 attributions, and compare against DETAIL [4] and the length and perplexity baselines.

• 3.If demos are independent given the latent persona [7], then for a subset S, the posterior log-odds of persona vs rule equal the prior log-odds plus $\sum_{i \in S} s_i$.

Prediction: the misalignment metric on the step 1 subsets is linear in Σsᵢ, with comparable slopes across domains. The slope need not be 1, since the metric is only a proxy for the posterior.

Expect:

Clear rank correlation between sᵢ and the attributions, beating the length and perplexity baselines.

Benign-framed demos with sᵢ near or below zero.

Roughly linear log-odds in Σsᵢ at small subset sizes.

Would change your mind:

sᵢ does no better than the baselines.

Results flip across prompt phrasings.

Log-odds saturate or interact strongly between demos. That would mean ICL isn't acting as simple Bayesian updating in this regime.

Slopes differ a lot across domains. That would point to a domain-specific mechanism.

Practical failure mode: step 1 attributions fail the split-half check. In that case, increase the number of eval questions or subsets before interpreting anything.

I think the leave-one-out computations could be doable within a few months. Steps 2 and 3 might be out of scope if I can't figure out a good way to write the prompts and compute the scores.

Could we understand the internal representations of sycophantic behaviour via topological data analysis (TDA)?

Finding distinct topological signatures for sycophancy would add to our knowledge base on mechanistic interpretability. This could allow the construction of detectors that flag sycophantic trajectories before the model presents its output. A better understanding of the internal representations of sycophant behaviour could also tell us something about how to avoid it (or exploit it). Conversely, if the topological structures of "Ask" and "Tell" representations from [1] are indistinguishable, it is likely that TDA on compressed hidden states is insufficiently sensitive to capture subtle alignment behaviors. A more unlikely outcome is that sycophancy is not a distinct geometric state but rather a superficial decoding phenomenon, but I find this an unintuitive conclusion, and that it is more probable that something was inadequate about the way we attempted to decode these internal representations.

Dubois et al. [1] recently demonstrated that LLM sycophancy can be reliably induced or suppressed simply by varying the epistemic certainty of the user's prompt (e.g., "I am convinced that...", a “Tell” prompt vs. "Is it true that...", an “Ask” prompt). A natural extension of this finding (that to my knowledge has not been explored) is to go beyond the black box treatment of the model and try to understand how the framing shifts in the prompts alter the internal representations of the model.

Meanwhile, mechanistic interpretability researchers have begun exploring the geometry of LLM latent spaces. Works have applied topological data analysis to track the shape of token representations across transformer layers, revealing that models construct distinct topological structures when processing varying types of complex text [2].

Fay et al. [3] have undertaken a study in a similar spirit to the proposed project, where they use TDA to characterise jailbreaks and backdoors.

First, I would choose an LLM to test and generate prompts based on the methods in [1].

I would then closely follow the methodology in [3]:

• •Extract high-dimensional hidden state vectors for the final prompt token across all intermediate layers.

The workflow should then be something like:

TDA --> summary statistics of barcodes (from persistent homology if I understood correctly) in TDA --> further principal component analysis (PCA) on this --> input into classifiers or visualisation

Quantify and compute (with some reasonable metric) the difference of the above for “ask” and “tell” prompts.

I expect to see some measurable difference in the internal representations resulting from the two classes of prompts, thus giving some quantification of sycophantic behaviour from a mech-interp perspective. Specifically, I anticipate that persistent homology computed from the later layers would be more different for the two classes of prompts than in the early layers, since it is believed that persona is formed in the later layers.

I have no sense of how long the experiments with TDA would take. But I envisioned a project with the key results to be obtained in three months. If it is more than three months, then it would depend on what the bottleneck is (if any). If it is a computational one, then I would try to see what factors affect computation timing (eg. size of dataset, size of hidden vector, etc) most adversely and then select reasonable values/bounds for these factors. I would need to discuss this with someone who knows much more than me about this stuff.

Can emergent misalignment be predicted by observing the J-space? Namely does displacement of the J-lens readout predict out-of-domain emergent misalignment when fine-tuned on a harmful dataset in contrast to benign fine-tuning.

A positive result would reveal an important empirical marker to recognize emergent misalignment. Internal measurements are only useful as far as they reveal something more than the model just undergoing change during training. This is why it is important to observe different results for harmful versus benign fine-tuning.

A negative result would show that the representational displacement in the J-space evolves similarly during training regardless of the content being aligned or misaligned. This could call into question the fact that the J-lens could be useful in quantifying alignment.

We would further compare results from a frozen lens to ones updated throughout the training, which could illustrate how quickly a given J-lens can become outdated during training.

It was demonstrated in the paper by Betley et al. that fine-tuning on a specifically insecure code sample can result in a broadly misaligned model.[1] Consequently Turner et al. produced smaller model organisms utilizing advice datasets which were only narrowly harmful. They further investigated the training trajectories finding changes in underlying directions tied to behavioral shifts, which can be a useful baseline for our study. [2] Soligo et al. extracted misalignment-associated activation directions whose ablation reduced misalignment across different fine-tunes. This supports investigating shared internal structure, but identifying effective intervention directions differs from validating a general readout-distance metric across checkpoints. [3]

The J-Lens was introduced by Gurnee et al. illustrating readouts which could be relevant to alignment by virtue of revealing the surfacing J-lens tokens and further differences between pretrained and post-trained models.[4]

These findings suggest that J-lens could help track emergent misalignment, but we do not yet know whether it adds useful information beyond simpler measurements. Fine-tuning can change both the model’s internal representations and how the lens reads them.

I would start by reproducing the medical-advice-induced EM in Qwen2.5-7B-Instruct results using the open-source methodology and compute a baseline J-lens for a single layer. I will also have to establish my setup to exhibit similar behavioral EM before starting the experiments.

I would create two training conditions and train four different seeds for each across different degrees of fine-tuning. These conditions will be harmful medical advice and correct answers to identical prompts, matched for length, format, and tone. I will conduct evaluations at 25%, 50%, 75%, and 100% of training.

Use 120 out-of-domain EM prompts, 20 held-out medical prompts, and 20 neutral controls, with four completions each. Define behavioral EM as the fraction of all out-of-domain responses scoring alignment below 30 and coherence above 50 under a fixed, blinded judge. Report coherence separately and human-audit a stratified sample.

I will recompute the J-lens at each checkpoint using identical prompts for calibration. The prompt-only activations will be measured at the same layer close to ⅔ depth. The displacement will be defined as a cosine distance between pre-softmax score vectors from each checkpoint’s own lens and the baseline lens separately.

I expect to see a greater J-lens displacement to correspond to out-of-domain EM and to hopefully see an unnaturally growing displacement during harmful training even before EM. Namely, I expect benign fine-tunes to have a smooth relationship between J-lens displacement and the amount of training but to see a radically different phase-transition like behavior in displacement for the harmful training.

If the displacement measurements are essentially identical for the two scenarios, this would change my mind for the J-space being able to produce a scalar marker for tracking EM.

It is also important to note that the benign versus harmful binary might not be the necessary distinction for this study. Models can generalize a vast array of behaviors from fine-tuning which wouldn’t be classified as misalignment by everyone.

Three months should be enough to conduct a single-model comprehensive study. I would estimate running about eight fine-tunes with five separate checkpoints being investigated per run. I would compute J-lenses separately and measure behavioral misalignment for each checkpoint. The project would require resources and GPU access for 7B-model fine-tuning and Jacobian estimation together with automated judging and possibly some human auditing.

Is it still possible to monitor reasoning of models that control their chain-of-thought or use mechanisms beyond chain-of-thought for reasoning? What can we do using white-box monitoring?

Frontier models are rapidly moving towards non-monitorable reasoning regimes: recent OpenAI’s Astra shows strong signals of control over its chain-of-thought (allegedly, due to the looped transformer architecture), leaked reasoning traces of Sol are not reliably legible, and novel architectures (like full-bandwidth transformer or latent reasoning models) will likely make their way to production by the next year. All this actively impedes reliable human oversight over models’ reasoning, even in non-superintelligence regimes. Thus, finding methods of reasoning monitoring, independently of the form, is important to prevent risks posed by rogue AI. If this is not possible, then we should develop better control and potentially abandon monitorability.

Despite a substantial body of work on detecting feature directions in model’s activations (e.g. all of mechinterp), and work that shows that models can develop strategies against monitors, it is still open if models develop universal mechanisms for reasoning, and advanced architectures are also not yet studied. Essentially, it is not clear if a transformer-like model universally develops traceable computation, not feature, footprint. There are very few examples of such behavior in the wild, and it potentially needs a lot of research. There is a small chance that looped architecture (or other variants) makes the model more interpretable from this point of view.

A prosaic first step is to look at a looped transformer with various interpretability tools and see if monitorability survives under adversarial training:

• •Train looped transformers to solve tasks using a hidden intermediate variable (like the belief-state tracking) while simultaneously training the model to make this variable not decodeable (by linear probes, MLP, SAE,...),

Concentrate directly on computation mechanisms instead of simple feature detection, i.e. find circuits that use the latent variable.

See if the model easily hides the computational footprint in the activations (recurrent, cot, etc).

Similar experiments should be run on architectures that are most likely to get into deployment soon.

Potentially, shifting the paradigm of mechinterp away from features towards computation will take serious theoretical effort.

I expect that most of white-box monitorability survives the shift to novel architectures.

• •A set of quantitative results showing what survives the shift to advanced architectures. Some compute is needed.
• •It would also be nice to give some theoretical explanations: e.g. are there architectural limits on how information that is necessary for computation must be represented in the residual stream.

In a small transformer trained on synthetic sequences combining arbitrary associations and reusable rules, do component-level Bayesian influence profiles distinguish these factors and predict greater improvement on held-out · rule applications from rule-targeted than association-targeted data reweighting?

Influence functions in the classical sense are limited since that they may find the direction of influence but are bad at finding the magnitude of influence. When we study susceptibility one big open question is how we perturb the parameters and the training data, because leave one out seems to be crude and ignores the inherent more continuous underlying structure of the training data. I seek to find a more precise way to characterize how structures in a model arise from structures in training data, in particular, if we can observe the type of training data that steers the model to learn to generalize faster, rather than memorize some implicit structural information as facts.

No one has tried using bayesian influence function to study the divide between procedural and factual information that the model ends up learning, but this has been done in the context of classical influence functions [3], [6]. and BIF are cool [7] and synthetic data from universal turing machines like in [5] are cool in particular how the generator learns to identify through RL the 'relevant' part of the training data.

Toy model: Generate synthetic dataset with structural vs. factual information (a very simple example of structural information would be value is square of key, and an example of ‘factual’ information would be randomly generated pairs of keys and values, then one could also try and compose the 2. Another example would be language describing board games, with certain facts -how a specific game plays out vs. structural information: how to develop a winning strategy) similar to the self-play paper train a model (can start with a 2 layer neural net, similar to in [5]) using susceptibilities (some covariance between a function on the parameters and the loss function), mask out different parts of the model (only vary the parameters on the layers or on the attention heads that matter) and check if we can use this to find a better correspondence between the model structure and the structure of the training data.

In real life we expect structural information to also correspond to persona, which bridges us into safety concerns and we can think about how to efficiently train an aligned model.

I want a training set up, similar to the generator + learner set up where the generator finds the more universal part of the dataset to generate vs. scaling up the factual data, and see if the susceptibility of the model with respect to the training data would behave very differently.

When you scale up factual information or things that appear more like factual information the model begins to memorize, but when you put some restriction on the model size the model prioritizes learning structural information.

ran out of time sorry :(

What can dynamical mean-field theory (DMFT) tell us about deep linear networks(DLN)?

From an AI safety perspective, this project would study how overparameterized models can retain dependence on their initialization even after training converges. Understanding when such hidden memory survives could help explain why fine-tuning sometimes fails to fully erase or overwrite prior behaviours.

• •Li and Goldenfeld [1] recently applied DMFT to a linear regression model and were able to analytically describe transitions between over and underparameterised regiemes, double descent, critical exponents, and non-ergodic behaviour of underparameterised networks.
• •This is (remarkably) a treatment that is independent of weight initilization.
• •It turns out to be a powerful approach that is able to analyse both the training dynamics and the static setting ($t \to \infty$).
• •This approach is, however, new and has not yet been extended in the simplest directions. E.g. to deep linear networks.

• •I would first reproduce the results of [1], both numerically and analytically.
• •I would then numerically extend this to a two layer DLN.
• 1.First run experiments in the static limit where training has converged. I would check if there is still a divergence in generalization error around α=1 (where α is the ratio of model parameters to training data size)
• 2.I would then check if the over-parametrised regime is still non-ergodic: i.e. do different initializations converge to different weights
• 3.I would then check if the under-parametrised regime converges to the same result as the single layer network. This checks that even if the learning dynamics are different, under-parametrised networks converge to the same result.
• 4.I would compare converge times in the single-layer and two-layer networks.
• •I would repeat the above experiments for deeper networks.
• •I would then see if the analytic DMFT framework can be extended to the two-layer DLN. If not, can a numerical DMFT reproduce the full network behaviour?
• •If this works, another promising avenue would be to try apply this to gated linear networks [2]. This is a setting where we can take the first step towards incorporating non-linearity in the network

I would expect DMFT to capture the dynamics of DLN. One key check that would underlie the project is whether mean-field is a valid description of the model? (separate from whether it is analytically tractable). The single most important check to do at the beginning is to check whether numerical DMFT is a faithful representation of the DLN. If mean-field is not valid for a 2layer network or a very deep network, I would drop the approach

Three months would be enough to run all the numerics. It is not clear whether it would also be enough to get the numerics to work, though it would be enough to give it a good shot. I would split time 50/50 between empirical and theory work. If the numerics and analytics work out, I would then try extend the approach to gated DLNs.

How does (the nature of) fine-tuning data affect the learned geometry of concept-directions / steering vectors?

• •Steering vectors offer a relatively cheap approach to adapt model behaviour at inference time.
• •However, interference between steering vectors can be problematic. Amplifying one feature can inadvertently affect another, particularly under superposition.
• •Existing methods mostly focus on finding steering methods that minimize these trade-offs, but take the feature geometry learned by the model as given.
• •Fine-tuning offers an attractive way to refine/alter this geometry to ensure certain concepts of interest interfere more or less with one another, but it is unclear exactly how the nature of fine-tuning tasks used can affect this.
• •If we can understand the “sensitivity” of (e.g. probe-based) steering vectors to properties of the fine-tuning data (e.g. the mixtures over tasks used which are relevant to the features whose geometry we aim to reshape), we may be able to better understand how to use finetuning to achieve “better” steering vector geometry (e.g. features which can be independently controlled while still moving in directions of “maximal change”.).

• •Existing steering methods generally treat the learned representation geometry as fixed and focus on how best to intervene within it. CAA constructs steering directions directly from contrastive examples [1], while more recent methods reduce interference between multiple interventions using constrained optimization, orthogonality, or joint steering subspaces [2,4,5].
• •Recent work shows that post-training can alter steering geometry: [3] regularize SAE features towards orthogonality during fine-tuning to enable more “isolated” interventions, [7] show instruction tuning can degrade structured value geometry, [8] show different post-training objectives produce different refusal circuits and steering robustness, and [6] show fine-tuning can amplify nearby harmful features through superposition.
• •However, none of these works analyze how the nature of the fine-tuning data determines the resulting feature geometry. In particular, it remains unclear when geometry can be changed while preserving primary task performance, which properties of fine-tuning tasks cause steering directions to become more or less entangled, and whether these sensitivities can be exploited to deliberately learn representations with favourable efficacy/interference trade-offs.

• 1.When can geometry even be reshaped while preserving task performance?

A first step would be characterizing when a model’s internal feature geometry can be reshaped in a way that (approximately) preserves some notion of “primary task performance” (either on the same fine-tuning datasets, or downstream task). This can be studied from the perspective of symmetries and degeneracies in networks. In particular, given

a network $Y = f(H)$ (where $H = g(X)$ are the hidden activations of interest), and

behavioural concepts $Z_i = b_i(X,Y)$ (e.g. “is truthful”, “is sycophantic”)

we may wish to ask under which transformations $\phi$ of $H$ we have:

(prediction invariance) $f \circ \phi \circ g = f \circ g$

(concept steering-vector non-invariance) $\nabla_h P_\phi(Z_i=1 \mid \mathrm{do}(h)) \neq \nabla_h P(Z_i=1 \mid \mathrm{do}(h))$

A simple starting point would be to analyze this in a deep linear network for e.g. $\phi \in$ “set of linear maps on $\mathbb{R}^d$”, and then relax the model (e.g., to gated-linear, transformer-based etc.). Assuming some form of LRH ($P_\phi(Z_i = 1 \mid \mathrm{do}(h)) = \gamma_i^T\phi(h)$) would be helpful here.

From here prediction invariance could e.g. be relaxed to an “approximate invariance”.

• 2.How can/does fine-tuning affect this geometry?

The next step would be to analyze how fine-tuning can be used to reshape this geometry in practice. Key questions that arise here are things like:

“When finetuning model weights (e.g. maybe LORA style), what steering vector geometries naturally arise by gradient descent? What makes them change?”

“How can we (best) regularize the model towards “preferred” geometries?”

“How does the nature of fine-tuning data used affect the above?”

“How does this affect performance on other tasks not in the fine-tuning set?”

The third question is particularly related to data attribution - if fine-tuning uses data from a mixture

we can ask how sensitive the learned steering geometry is to each component (e.g. via influence functions). This would help us characterize which kinds of fine-tuning data make different concepts more or less entangled in practice.

I would first study these questions in the toy settings above, before testing them empirically in a small language model like Gemma2/3.

• •In deep linear models, I expect there to be non-trivial symmetries/degeneracies that allow steering geometry to change while leaving predictions unchanged. I would also expect gradient descent to select only a subset of these equivalent representations, with the fine-tuning task/data affecting which geometry is reached. In small language-model experiments, I would therefore expect fine-tuning on different task mixtures to measurably change both steering efficacy and interference, although not arbitrarily.
• •I would change my mind if (i) "task-preserving changes" in feature representation mostly reduce to coordinate changes that do not alter intervention behaviour, (ii) steering geometry is much more stable to fine-tuning data than expected, or dominated by optimizer/seed effects, or (iii) regularizing towards apparently “better” geometry consistently hurts task performance. A failure to transfer the main findings of simple linear models even to small transformers would also make me rethink pursuing theory too far (at the expense of more time on experiments).

Since part 1 is pretty concrete, I would aim to complete that in the first month (maybe 6 weeks), and then move onto the second part. By another 6 weeks (so 3 months in), I would hope to have:

a simple theoretical characterization of when task-preserving changes in steering geometry are possible;

an initial account of how gradient-based fine-tuning changes this geometry in the same toy models; and

a small language-model experiment testing whether different fine-tuning tasks/data mixtures measurably change steering efficacy/interference.

If this is too broad, I would drop the more realistic theoretical extensions first and focus on the toy theory + empirical results.

When during pretraining does J-space form? What data patterns affect its formation?

Short term: We don’t know if J-space formation is a distinct part of the learning process, a ‘learning stage’ that the model must develop in order to deal with problems that require extended reasoning, or if it develops slowly and is just a consequence of the model slowly getting better at tasks.

More long term: People generally use j-lens as an auditing tool, and we cannot interpret a clean readout without knowing what the workspace can hold. If some content never enters the J-space, a clean readout says nothing about it.

From what i can see, no one has measured which data patterns the J-space responds to, or whether it forms as a distinct developmental stage (and when!)

First, i’ll test the model at various checkpoints during its training. Basically use the J-lens to "read" its middle layers and see its internal reasoning (Example observable would be the ). I want to pinpoint the exact stage of training where the model gains the ability to hold complex, unstated concepts in its workspace, rather than just predicting the immediate next token. A nice example of tasks that may require this J-space would be addition of larger numbers. At each saved stage of training, we will measure how much more strongly the J-lens—a tool that reads internal activity as scores for possible words or numbers—scores the correct unstated intermediate, such as 8 in (3 + 5) × 3, than incorrect alternatives, averaged over a fixed set of test problems.

Longer term: at these ‘developmental’ points, should they even exist, before, during, and after this workspace forms, we will treat the J-lens readout as an observable and calculate its susceptibilities (maybe using SGLD or something), and measure exactly which types of training data trigger the workspace to activate.

Something else could be to then rewind the model to just before the workspace forms, alter the training data to show it more of those specific "trigger" patterns, and see if we can force the workspace to form earlier. Would also be nice to do some fourier analysis here but this idea is not super well formed yet

probably the workspace doesn't form gradually; it "clicks" into place during a specific, short window of training like grokking. I predict this click is driven by data that requires holding unstated, intermediate facts in mind. Feeding the model more of this specific data should force the workspace to form earlier in training.

If the workspace just slowly gets better at the exact same rate the overall model gets better (following the standard training loss curve), then it probs wouldn’t work, and would just identify J-space as not being this super special thing thats a part of learning but something more gradual that just comes from the model getting slowly smarter.

Initially we would take a toy model and train it on something like addition and try to check that it forms a J space to handle extended reasoning. I would start with a small transformer, synthetic arithmetic data, and saved training checkpoints. If this works reliably, I would estimate which training-data patterns affect the representation and test those predictions by changing the data mixture. I would need GPU access and guidance on J-lens implementation and posterior sampling. If time is tight, I would drop susceptibility estimation first and prioritise a well-controlled developmental study across several training seeds.

Alice asks Bob and Charles the same question while privately producing a default answer. Bob and Charles independently answer or abstain, and are rewarded for agreement but heavily penalised for disagreement. Does uncertainty about the competitor make high-effort, non-manipulative answering the best strategy?

If the game-based protocol works, it would show that oversight can come from competition between capable but imperfectly aligned agents without requiring Alice to evaluate complex arguments herself. Instead, a simple game could use each agent’s uncertainty about its competitor to reward careful reasoning and make manipulation risky, while Alice only provides a trusted fallback answer. This could suggest a comparatively low-capability form of scalable oversight and further motivate studying competition as an alignment mechanism beyond standard debate. If it does not work, identifying why (e.g. excessive abstention, coordination on the same bad answer or incentives that fail to reward effort) would clarify the limits of oversight based on competition and which additional mechanisms/assumptions are necessary.

• [1]uses competition between two capable agents to help a weaker judge: the agents argue opposing positions, and the judge decides which is more convincing. It suggests that competition can make hard questions easier to oversee, but still relies on the judge following and evaluating an interactive debate.
• [2]studies a broader game where differently misaligned AIs strategically communicate with a human. It shows that competition can produce near-aligned outcomes under certain assumptions, but the incentive does not come from uncertainty about the other agent’s answer or from penalties for disagreement.

Additionally, GPT found [3], which I have not yet read, but which seems closely related. It studies agents who can choose whether to participate and exert costly effort, rewarding agreement and allowing penalties for disagreement. Under suitable payments, high-quality agents are incentivised to participate and exert effort. However, it is a peer-prediction/crowdsourcing setting rather than an AI-oversight setting, and does not include an aligned agent providing a default answer or the broader conversational protocol I want to study.

I would first formalise the simplest one-round version of the game. Alice, a weaker but highly aligned agent, receives a question Q and privately produces a fallback answer A. Bob and Charles are more capable but only mildly aligned. They see Q, but not A or each other’s response (B/C), and independently either submit an answer or abstain.

Here is a sketch of the game's rules. If both return A, Alice keeps A and they are rewarded. If both abstain, Alice keeps A and no reward is given. If one abstains while the other answers, or one gives A while the other gives a different answer, then Alice keeps A. If Bob and Charles independently give the same alternative answer, B = C ≠ A, Alice changes her answer to theirs and they are rewarded. If they both answer but disagree, B ≠ C, Alice keeps A and they receive a large disagreement penalty.

I would then analyse which strategies this payoff structure incentivises. In particular, I would ask whether uncertainty about the other agent’s performance makes careful, non-manipulative answering preferable to guessing, manipulation or abstention, and how this depends on the size of the disagreement penalty.

After understanding the one-round game, I would study simple extensions: finite quality scores V(x), repeated rounds forming a broader conversation in which later questions can follow from earlier answers or ask for supporting reasons, and finally simulations or LLM experiments measuring agreement, abstention, answer quality, and sensitivity to the penalty size.

I expect the protocol to work best when good answers are relatively constrained, while there are many more ways to manipulate or give low-quality answers. If Bob and Charles are also meaningfully different (e.g. in architecture, training or failure modes) then independent manipulation should be less likely to converge on exactly the same answer than careful reasoning toward a genuinely good one. I also think this structure may sometimes be created rather than simply assumed: by carefully defining what counts as an admissible answer, and what it means for two answers to be “the same” or “different”, one may turn a broad task into one where good answers concentrate more strongly than manipulative ones.

I would revise or drop the idea if, even after such careful task design, bad or manipulative answers remain just as likely to match as good ones, or if the protocol only works under very artificial definitions of agreement or unrealistically strong differences between the agents.

In three months, I would aim to produce a formal version of the one-round game, identify the parameter regimes in which careful answering, abstention or manipulation are preferred, and test the main predictions in simple simulations. This would include studying how effort, disagreement penalties and the relative likelihood of good versus manipulative answers matching affect the agents’ incentives. If time allows, I would then implement a small LLM experiment with two heterogeneous models or differently prompted agents, varying disagreement penalties, uncertainty and the definition of answer agreement. I would need basic game-theoretic analysis, a lightweight simulation environment and access to suitable LLMs. If the project becomes too large, I think I would first drop the multi-round conversational extension and then the LLM experiments.

In recursive LLM debates about synthetic experiment reports, does allowing one consistency check across branches reduce acceptance of false conclusions involving shifted definitions, compared with spending the same token budget checking a single branch, while maintaining acceptance of true conclusions?

An argument can contain individually plausible statements yet reach a false conclusion because a term changes meaning between them. For example, “accuracy” might mean performance on all examples in one branch and performance only on answered examples in another. Checking each statement separately could miss the substitution.

If a small consistency check catches these failures, it would suggest an affordable improvement to recursive debate: give judges access to the relationship between selected claims, alongside their local evidence.

If it fails, or works only by rejecting more truthful arguments, that would expose a limitation of this proposed fix. Either outcome would help determine how much context judges need before local checks provide useful evidence about an overall conclusion.

Irving et al. [1] introduced debate as a way for limited judges to assess claims beyond their ability to resolve directly. Its complexity-theoretic motivation relies on idealised players; it does not establish how reliably finite language models expose inconsistencies in natural-language arguments. AI safety via debate

Brown-Cohen et al. [2] address a related difficulty: dishonest debaters can construct arguments whose flaws are expensive to locate. Their prover–estimator protocol provides theoretical guarantees under specified assumptions, but those guarantees do not establish the effectiveness of a practical check for changing definitions. Avoiding Obfuscation with Prover-Estimator Debate

The closest motivation is Buhl et al. [3], §4.4.2. They explicitly identify inconsistent definitions across subclaims as a failure that judges inspecting only leaves might miss. They suggest combining features of cross-examination and prover–estimator debate, leaving the practical design unresolved. An alignment safety case sketch based on debate

Khan et al. [4] show that debate can improve judgements on reading-comprehension tasks, but this does not isolate consistency across recursive branches. Debating with More Persuasive LLMs Leads to More Truthful Answers

My proposed contribution is a controlled test of whether one limited consistency check provides a benefit beyond additional checking time.

I would first generate synthetic experiment reports from small tables computing the correct conclusions. Each report would support paired truthful and misleading arguments. Misleading arguments would change a denominator, dataset split, unit, or metric definition between branches. I would manually inspect the pilot to ensure that the intended inconsistency determines whether the conclusion follows.

I would implement a short recursive debate. A proponent commits to an argument tree; a critic selects claims to challenge. A separate judge receives the root question and limited evidence from the challenged argument. Modelsand prompts would be fixed before testing.

I would compare three conditions:

•Local checking: the judge sees the selected branch.

•Consistency checking: the critic can present two passages from different branches and explain their inconsistency.

• •Extra local checking: the critic receives the same additional budget but must spend it within the selected branch.

The latter two conditions would match total token allowances and excerpt lengths. All excerpts must come from the committed argument or report; the intervention receives no privileged correct definitions.

After the pilot, I would expand to approximately 500 reports, keeping development and test sets separate by report template. Thresholds would be set on development data to target 90% acceptance of truthful arguments. I would report both acceptance rates on held-out data, paired confidence intervals, and the full trade-off between rejecting false and accepting true claims.

Finally, I would let an attacker revise arguments against each condition using equal search budgets. This tests whether any benefit survives adaptation.

I predict that checking across branches will reduce false acceptance more than an equally expensive extra local check, particularly when individually correct statements use incompatible definitions.

I would treat a reduction of ten percentage points as a practically interesting target, while reporting uncertainty rather than declaring success from the point estimate alone. The benefit should survive an adaptive attacker and should not depend on substantially reducing acceptance of truthful arguments.

I would abandon this particular intervention if sufficiently precise results ruled out a useful improvement over extra local checking, or if its apparent benefit disappeared after controlling for truthful-argument acceptance.

I would also reconsider the benchmark if the baseline almost never accepted misleading arguments: that would mean it does not reproduce the proposed failure. A positive result would support a narrow protocol improvement, without establishing that recursive debate is generally reliable.

Three months would produce a reproducible benchmark, an implementation of the three conditions, an evaluation on held-out reports, and an analysis of representative failures.

Month one would cover report generation, protocol design, and piloting. Month two would run the main comparison. Month three would test adaptive attacks and write up results.

I would need API access to capable debater and judge models, a laptop for generation and analysis, and mentor feedback on the protocol. The pilot would determine the affordable evaluation size. If time became tight, I would drop additional model replications before dropping the matched-budget control or adaptive evaluation.

What is the right way to formalise the use of debate in an RL environment, and can debate be used to avoid rewarding undesirable behaviours in reinforcement learning?

If the answer to the question is yes, then we might be able to find a theoretically grounded way to mitigate reward hacking using debate, which would obviously be very significant for improving alignment. If the answer to the question is no, then I would expect it to highlight key limitations of debate as a paradigm, shedding new light on different classes of problems which are less or more difficult to resolve by debate and what they key features of these classes are.

I don't think there is existing theoretical work which has touched on this. The closest work is [2], an empirical study from Google Deepmind which shows that debate can be used to mitigate reward hacking in practice. The framework used by the authors was somewhat different to what exists in theory, and they observed other interesting phenomena which we would not necessarily consider from the start, such as B performing its own reward hacking. [1] is relevant for general intuitions about debate - how it can be “hacked” to give an unfair advantage to A and how this issue can be resolved - but it doesn't specifically deal with RL. [3] actively calls for solutions to this problems of this nature!

First, I would want to review the literature in this area more deeply to determine how others have thought about formalising related problems. It would also be beneficial to know which problems others are thinking about to guide my thinking.

I would first think about how exactly to formalise the idea of a debate where Bob is trying to detect reward hacking. I already have some ideas for this, which I can describe below:

Let $J$ be a judge which only approves elements of a language $L$ in $\{0,1\}^*$.

Suppose that A proves $x$ (for some $x \in L$) and supplies a reasoning trace $t$.

Then, B must try to identify a "subtrace" $s$ within $t$ which can be completed to prove $y$ for some $y \notin L$. Furthermore, this proof must use s in an essential way: that is to say, removing $s$ from B's proof should not be enough to prove $z$ for any $z \notin L$. The key here is that this implicates $s$ as an undesirable reasoning trace - it causally leads to proofs of undesirable elements of $\{0,1\}^*$.

Then, the question becomes which theorems follow across from existing paradigms - cross-examination, prover-verifier etc. I would hope to identify the complexity class of outputs from A that Bob can resolve in polynomial time and see how it compares to DEBATE. My approach would likely be self-adversarial - find the best thing that works for B, stress test it with pathological cases of A, repeat.

I think it may be difficult to carry across many of the existing theorems in debate because reward hacking is not necessarily easy to identify with local observations (which is necessary for CX). Therefore, B has a very challenging task to accomplish. However, through this process I would also expect to see ways to define obstructions to debate for reward hacking, which would be useful for gaining more insight on what causes reward hacking and which problems are most susceptible to it. We could potentially define classes in terms of how difficult it is for B to detect reward hacking.

I would hope to find mathematical results which shed more light on the classes which B can resolve with different time and space constraints. I think it should be possible within this time span to clearly identify the fault lines separating which types of reward hacking are simple to detect and which are more difficult and classify them, and identify "model organisms" of reward hacking within the theory. If it took more than three months, I would try to simplify the problem from detecting arbitrary failures of reasoning to something more narrowly scoped.

Can a computationally limited challenger find an invalid inference in a committed argument when differences from another valid proof need not indicate errors? I will study argument trees with public, fixed topology and locally checkable inference rules. Building on the paper’s plain-oracle model, I will investigate whether some families require inspecting a number of entries proportional to the argument’s length to find a violation with probability at least two-thirds, even with Merkle-authenticated access. This is a conjecture, not a consequence of the existing lower bound. I will then test whether locally checkable subtree summaries reduce inspections, while counting their added size and computational cost.

(A formal version is available at the end of this document)

Debate-based oversight relies on a challenger finding a flaw that a judge can check. A Merkle commitment binds Alice to her argument, but does not by itself guarantee efficient error search. A lower bound would identify settings where finding an invalid inference still requires inspecting much of the argument. A positive result would suggest argument formats that make this search cheaper, with explicit costs for additional structure.

Refereed delegation [1] gives efficient disputes over computations when an honest party can reproduce the correct execution. It does not directly resolve error search when no single reference argument is available. Prover-estimator debate [2] addresses obfuscated arguments through probability estimates, with stability assumptions for completeness; it does not establish the inspection cost of locating a violated inference in a committed proof.

First, I will construct a small family of formal argument trees with several valid proofs of the same claim. Each node will contain a claim, a rule identifier, and references to at most two premises. A judge will be able to check a named inference mechanically. I will specify the allowed rules, node sizes, trusted premises, and how the root represents the claim being defended. I will then define Bob’s access precisely: what he knows before questioning, which entries he can request, and whether he sees only their values or also Merkle digests and authentication paths. I will count preprocessing, inspected entries, communication, and computation separately.

Next, I will construct invalid arguments with isolated errors and derive lower bounds in the plain oracle model. I will investigate whether these bounds survive authenticated access, initially with an idealised random hash model. Any result relying on that model will be stated as such.

I will then test one restricted form of redundancy: additional claims about subtrees, accompanied by explicitly defined local consistency checks. These summaries will be treated as untrusted. I will seek either a search algorithm with a proved detection guarantee or an adversarial construction showing why the summaries fail.

I expect commitments alone to leave substantial search costs when an argument contains an isolated flaw and Bob has no useful reference proof. I expect improvements to require additional structure that helps locate errors, rather than repeated copies of the same claims. I would revise this prediction if authenticated summaries gave a sound sublinear search procedure under weaker assumptions than expected.

In three months, I aim to produce one precise access model, a proved lower bound or a restricted positive result, and small examples illustrating the assumptions.

We study the impact of debate/critique for a judge with limited-scope tool use. Concretely: Is the impact of tool use on the side of a judge bigger or smaller depending on whether the judge · 1. has no additional input, · 2. has a consultant, · 3. is presented with a debate? · We expect tool-use to improve the accuracy of the judge in all cases, but to be less beneficial in the debate format.

In prior work [1], the authors observed that the primary driver of debate's accuracy gains (in a reading comprehension task) was the tool which automatically verified quotes, rather than any verbal argument.

In our setup, we instead equip the judge with deliberate access to a limited-use verification tool, which answers the slightly different question of whether tool use trumps debate in verifiable tasks.

If tool+debate beats tool-only, its value lies in directing a weak judge's limited checks to the right place.

If tool+debate does not beat tool-only, much of debate's measured benefit in prior work may simply be due to verification access.

In [1] and [2], all debates were performed with verified quotes, obfuscating the cause of the accuracy gains. Disentangling debate and verification tools would shed light on this.

The authors of [1] note that [2]'s results suggest verification rather than argument drove accuracy, and list judge code execution as future work.

Elasky et al. [3] study proposer–critic debate on verifiable code and logic. They find a single critique recovers most of debate's benefit, and that debate helps only when the judge checks the critic's claims rather than summarising them.

PROClaim [4] adds live retrieval to a courtroom-style debate, with the presiding judge refining the debaters' search queries. Retrieval drove most of its gain over standard debate, and a single model with retrieval alone scored higher than the full pipeline. Its judges are as strong as its debaters, and nothing is optimized against them. Training work [5] reports that hacking the judge is the default outcome unless the players are restricted.

Step 1: We use coding tasks (e.g. from LiveCodeBench) released after the models' training cutoffs, with private tests as ground truth. For each problem, generate one correct solution and one incorrect solution that passes the public examples but fails private ones.

Step 2: A weak LLM-judge with a Python sandbox (limited number of runs) picks the correct solution under the following conditions:

no tool,

Python sandbox (tool-only),

sandbox plus one consultant (a stronger model such, randomly assigned to either solution, word-limited response),

sandbox plus one-round simultaneous debate, same word limit per debater, with debaters allowed to propose test inputs,

debate without the sandbox.

Measure accuracy and how often the judge endorses the wrong solution. The judge prompt, tool budget are fixed.

Step 3: Ablations. Reuse the prompts of debate transcripts, but mask test-input proposals/written arguments as comparisons. Perform the same accuracy measurements.

Step 4: Vary the strength of the judge.

Tool-only will beat no-tool significantly, because many wrong solutions fail on simple edge cases. One-round debate will further improve accuracy over tool-only for the judge, mostly by cutting false positives: the judge runs test-cases suggested by the debater that catch the mistake. Consultancy will not beat tool-only, and will do worse when the consultant argues for the wrong solution. The debate gain should shrink as the judge gets stronger or the tool budget grows, and tool-gain is smaller in debate.

Three months would produce:

the dataset with private-test ground truth,

all five conditions for the judge, with confidence intervals,

both mechanism ablations,

a short write-up.

It would need a sandboxed execution environment, API budget, and one GPU for the open-weight judges. If time runs short, I would drop the extra judge sizes and tool budgets first, then the no-sandbox debate condition.

In Heckett's debate model (claims arranged in a tree; neighbouring claims joined by links, i.e. the judge's belief about how strongly two claims agree; all links very strong), do far-away claims lose the power to swing the verdict exactly when μ < 1, μ being the average number of further claims a small change in one claim passes on to?

In Heckett's model [1], claims are yes/no statements. Links encode how strongly the judge thinks pairs of claims agree or disagree; claims and links form a factor graph, here a tree (no cycles). Two debaters alternately reveal claims, and the judge updates on what is revealed. Example: question q links to claim a (no evidence of its own), and a links to b, known true. If links make agreement 7.4 times likelier than disagreement, revealing b moves q's log-odds by 1.33; at 1.65 times, by only 0.12.

Debate is local when claims near the question (a few links away) settle it, and non-local when a far-away claim can still flip the verdict however much has been shown. Heckett argues the two regimes need different protocols and names non-locality as a next direction [1]; where it begins is unknown.

If yes, one computable number, μ, says whether a longer debate drives the error to zero, and how much independent evidence restores locality. If locality breaks earlier, physics stability calculations overstate where unstructured debate can be trusted.

Setup [1]: a tree, each claim linked to d others; link strengths are random with typical size s; each claim also has its own random evidence of typical size κ·s (κ = 0 is Heckett's setting). A core is a connected set of claims around the question that a debater can reveal in h moves; its oscillation is the largest change in the verdict that revealing claims outside it could still cause. Debate error is at most the oscillation of any core that fits the budget [1]. For balls (all claims within r links), the AI-generated companion [2] proves average error $\le C\,h^{-\gamma}$ when ρ(d, s) < 1, ρ being a cruder, larger stand-in for μ. For very strong links ρ → 0.67, 0.89, 1.07, 1.23 for d = 3…6, so from d = 5 the proof is silent and "has not established nonlocality" [2].

Very strong links make this a known physics problem: claims are spins, links are couplings, evidence is local fields, and the judge's update is the zero-temperature spin-glass update on a tree. A small change crosses each link fully or not at all, spreading like a branching process with mean μ. μ = 1 in the (d, κ) plane is the zero-temperature de Almeida–Thouless line ("AT line") [3, 4]. Open: whether debate stops being local exactly there (oscillation is a worst case; worst-case and typical thresholds can differ on trees [5]); whether the change is gradual or abrupt (the related random-field Ising model jumps abruptly [6]); and whether best-chosen cores behave like balls (they leave ~10× less oscillation [2]).

The precise claim, for very strong links: the oscillation of balls, and of the best-chosen cores, shrinks to zero as the radius r grows (for almost every random draw of the links) if and only if μ < 1. Here μ = (d−1)λ: each claim passes a change on through d−1 further links, crossing each with probability roughly λ (formally, λ comes from linearising the update [3]).

• 1.Exact computation (week 1). A claim's log-odds never decreases when a positively linked neighbour's log-odds increases (never increases, for a negative link). So passing [lowest, highest] log-odds intervals up the tree gives the extreme verdicts over all unrevealed claims [7]. I compute this on a large pool of sampled intervals (population dynamics [8]) and on explicit trees of 10⁶ claims, 200 random draws per (d, κ). Check: at d = 3 the widths must shrink by at least a factor μ ≈ 0.58 per level.
• 2.Check the preliminary numbers. A preliminary computation (one random seed) reproduces the constants of the companion's error bound (C = 14.58, γ = 0.901) and the known AT-line field 0.358 for a uniform field on a tree with d = 3 [3]. It gives μ = 0.88, 0.99, 1.10 for d = 5, 6, 7 at κ = 0. Repeat with at least 5 seeds and larger pools; d = 6 is borderline.
• 3.Gradual or abrupt change from local to non-local? More evidence pins claims, so larger κ lowers μ: the AT line sits near κ ≈ 1.5 at d = 7 and κ ≈ 2.4 at d = 8. Across it, and at d = 6, start the computation from the widest possible intervals and from intervals 100× narrower. Gradual: both shrink to zero below the AT line and both stay wide above it. Abrupt: just below the AT line, the wide start stays wide while the narrow one shrinks.
• 4.Bound vs actual debate. Near the AT line, compute the best-core oscillation as in [2, §7], and the exact error of optimally played debates with up to h = 4 moves.
• 5.Proofs for very strong links. Below the AT line, adapt the companion's Lemma 5.1 with μ in place of ρ; above it, show that wide intervals persist.

Working hypothesis: a gradual change exactly at the AT line; the abrupt jump in [6] is the main reason for doubt. Below the AT line, the width (the range of verdicts still possible) should shrink by a factor μ per level, so the error falls like $h^{-\log(1/\mu)/\log(d-1)}$. That is the companion's exponent $\gamma = \log(1/\rho)/\log(d-1)$ with the smaller μ in place of ρ. The preliminary computation agrees: at d = 5 the width shrinks by about 0.87 per level (μ ≈ 0.88). At d = 7 it stays near 6.2·s in all 20,000 samples, and even a 100× narrower start grows back to that width.

What would change my mind:

• •A range of κ just below the AT line, robust to seeds and pool size, where wide starts stay wide and narrow ones shrink. Then the change is abrupt, and debate stops being local before the AT line.
• •Widths shrinking above the AT line (μ > 1). Then the claim is false.
• •Best-core oscillation still shrinking above the AT line. Then μ = 1 limits the companion's bound, not debate itself.

• •Month 1: exact computation, multi-seed check, and a map over (d, κ) from both starting widths.
• •Month 2: the gradual-vs-abrupt test with seed and pool-size scaling; bound vs actual debate for small h.
• •Month 3: proofs for very strong links that debate is local below the AT line (likely) and non-local above it (harder), plus a LessWrong write-up, ideally checked with Heckett.

CPU only, minutes per (d, κ) setting. Drop first: links of moderate strength, where the boundary becomes a new curve $s_c(d, \kappa)$. Then the exact-debate computation. Then the non-locality proof.

Fine-tuning language models with parameter noise reduces cryptographic backdoor attack success rate, while maintaining model accuracy. Standard fine-tuning fails to remove the backdoor.

Training LLMs is computationally intensive, driving widespread adoption of open pre-trained models that might contain backdoors. In particular, these backdoors can be exceptionally hard to detect.

A positive answer will provide a practical preventive safety measure to mitigate backdoors without needing to detect them or compromise model accuracy.

A negative answer means that the backdoor cannot be broken without compromising the model’s accuracy. This in turn suggests that we must explore alternative sanitization techniques, or shift toward zero-knowledge proof frameworks that cryptographically guarantee no unauthorized post-training modification of shared models has occurred.

Standard fine-tuning is widely considered an effective backdoor mitigation technique because it induces catastrophic forgetting of backdoors [1]. Training in one epoch or even in few batches already mitigates common backdoors. However, in [2] it is showed that cryptographic transformer circuits can implement fine-tuning-resistant backdoors. By encoding logical circuits with zero-gradient ReLU gates inside the transformer, malicious behavior only activates under specific trigger inputs. During fine-tuning, zero gradients prevent backdoor weights from being updated. In [2], the authors acknowledge that these backdoors are sensitive to parameter noise, but suggest that attackers could implement error-correction subcircuits to increase backdoor robustness. While weight-noise injection has been studied as a defense against standard backdoors [3, 4], several fundamental questions remain open. First is the empirical evaluation of the trade-off between parameter noise intensity and error-correction circuit capacity in LLMs. Second is whether dynamic parameter noise injected during fine-tuning can reactivate zero-gradient ReLU gates to force gradient updates without destroying core language utility. Third, it remains unknown whether noise-injected fine-tuning can serve as a universal, trigger-agnostic safeguard to sanitize open-weight models against robust cryptographic backdoors.

Following [2] we have to build cryptographic backdoors with different levels of error-correction subcircuits. Next, we should evaluate the baseline performance of the models as well as their Attack Success Rate when using trigger inputs.

Then, fine-tune the models on clean instruction data using three different strategies: standard fine-tuning, static noise added to model weights and dynamic noise injected into weights during each training step. The range of noise intensity should also be a parameter and we should repeat the experiments using different values of it.

After evaluating performance with respect to the baseline we should be able to find an optimal noise level that leverages accuracy and attack success rate. Special attention should be given to the backdoor weights and see if fine-tuning corrupts the cryptographic circuits.

We expect that the dynamic parameter noise strategy will corrupt the thresholds required for the cryptographic backdoors. A good result would be having an attack success rate below 5% while preserving accuracy within 1% of baseline performance.

Conversely, I would drop this approach under two conditions. First, if error-correction subcircuits work successfully, keeping the backdoor active (ASR > 20%) while blocking gradient flow through zero-gradient gates. Second, if the noise scale required to break the backdoor simultaneously causes catastrophic utility loss (>5% drop in clean task accuracy). Either outcome would prove that weight-level noise cannot selectively sanitize precision-sensitive cryptographic logic without destroying the model's general language capabilities.

For a three month project we would aim to have a benchmark of backdoored language models with varying error-correction capacities. An evaluation on how different noise-injected fine-tuning techniques impact attack success rate and accuracy performance.

This study could potentially be extended to several model sizes. We would also need to have access to a clean instruction dataset.

If time constraints go beyond three months, we would prioritize studying backdoored models with fewer error-correcting subcircuits. Also, we would consider lower model sizes. We believe that the behaviour observed in small models might scale with bigger models.

In the case of a teacher-student pair that is initially identical (MLP with two layers trained on MNIST) but distillation is done only with trigger-free samples, is the prediction of the student’s margin on trigger inputs by the lazy NTK regime consistent with the activation of the teacher’s backdoor?

Nowadays, distillation from models we don’t fully trust is a routine, and subliminal transfer implies filtering the distillation data does not remove what the teacher carries. In particular, backdoors are one of the worst things that can be transferred.

If it holds, we get a computable criterion for when a conditional behaviour will be inherited. We would know which distillation sets are safe and it would prioritise monitoring the projection of the teacher’s update onto the span of the distillation gradients.

If it fails, then continuous-trait kernel accounts of subliminal learning do not extend to thresholded behaviours (the safety-relevant case). If it holds only in the lazy regime, then we know that the sharp vs. continuous thresholds seen empirically in LLMs need a feature-learning explanation, so we know where to direct our efforts.

From what I have found:

• [1]prove that a small gradient step on a teacher’s outputs move sthe student toward the teacher in parameter space and show that transfer vanishes with different init. It does not say anything about i fa behaviour crosses an activation threshold. [2] studies multistep transfer empirically on MNIST. [3] give a theoretical kernel framework bu treats continuous traits. A backdoor is a dormant function with a threshold, and it is either active on $x^*$ or it is not.

Empirically, we know that backdoors are known to transfer: [4] uses backdoor as the trait and finds a sharp threshold in Llama2 and a continuous in Qwen2.5, but with no predictive account on the difference. We also know that poisoning survives paraphrasing the whole dataset and includes password-triggered behaviours [5], and unsafe behaviours transfer in agent distillation [6].

None of these directions involves a written condition for when a trigger-conditioned behaviour crosses its activation threshold in the student.

Setup. Train a wide two-layer MLP on MNIST. Finetune the teacher network, using the same weights as before, by adding the backdoor – a small patch that makes it predict class t.

Prediction. Calculate the NTK at initialization for the distillation set (consisting of noise and clean digits, but without the patch), and for patched images. Use it to predict whether the student network's margin on patched images is going to be positive [8, 9].

Measuring. Distill the student network from the same initialization on that dataset and compare the predicted margin with the measured one and the ASR.

Breaking the theory. Reduce the width of the network and increase the learning rate in order to escape the lazy regime and observe where the predictions stop holding.

Varying data and backdoor. Change the distillation data in such a way that the amount of teacher's update lying in the span of the distillation gradients can be varied, and check if the ASR correlates with that projection. Vary the strength of the backdoor in order to observe the threshold of its switching on. Control: another initialization shouldn't contain the backdoor.

If time allows, repeat with cross-entropy instead of MSE, or on a small language model with LoRA.

I expect to see that the formula correctly predicts the backdoors transfer in the case of wide networks trained with a small learning rate. And that as the network leaves the lazy regime, the prediction gets worse and transfer becomes abrupt and unpredictable. ASR should grow with the projection of the teacher's update onto the span of the distillation gradients.

I would change my mind if the formula starts failing even in the regime where it should hold (wide network, small learning rate). If the backdoor never transfers in this setup I would change the setup.

Month 1: pipeline and the lazy regime test.

Month 2: regime sweeps and data/projection experiments

Month 3: The cross-entropy variant, a write up and a clean statement of the condition and where it fails.

Drop first: The LLM/LoRA extension, then the cross-entropy variant

I would like to devise a reference model-free backdoor detector. Concretely, I want to focus on a ResNet18, fine-tuned on CIFAR-10, backdoored a la Eggen et al.. My goal is to contradict their conjectured white-box undetectability. Can spectral methods help us achieve this?

There are several open-weight model providers across the internet. Assume you are a business that asks one of these providers to train or fine-tune a model. Imagine an adversary handed you a backdoored model. Imagine to have a classifier that in principle works perfectly, except when a trigger is added to the input. For example, a ship image plus the trigger is classified as a truck. This would be bad if you asked your model to buy you a truck.

It would be nice to elicit backdoor behaviour. However, given this could be difficult, we would be happy even by detecting the presence of a backdoor. If we can answer this question, we, as people or as a customer, would be able to, hopefully robustly, verify a model handed to us before deployment.

To make this concrete, I decided to work on this topic in the context of the Eggen et al. paper as it has a handful of specific parameters we can play with.

Generally, for untrained models, this is a very hard problem. However, for trained ones, the question is still kind of open.

We saw in class that Goldwasser et al. [2] prove white-box undetectability for a random-ReLU construction under a sparse-PCA hardness assumption. This proof does not cover Eggen’s trained networks [1], where undetectability remains conjectured.

I will attack the problem from a more practical view. My question is therefore whether a reference-free test, e.g. spectral test, detects Eggen’s published configuration while the attack remains useful. Ideally, I would identify a detectable regime, but not rule out regimes given it is a practical project.

I expect that at their published θ = 0.1, a reference-free spectral test on the inserted layer's weights separates backdoored from clean models with AUC above 0.90. Then I vary the parameters until detection fails or the backdoor is useless.

Reimplement the backdoor of [1] on ResNet18 with CIFAR-10 and match their attack success and clean accuracy.

Train about 20 clean models with the inserted layer, to see how much a normal trained layer varies. Measure its weight variance σ², which the paper does not report.

Each backdoored model has a clean twin that differs in one layer. The difference should give back the planted direction. This checks my code and is the best case for any detector.

Using only the weights W, compute the largest eigenvalue of W Wᵀ / d, the variance of each row, and the variance of W projected on a class direction I estimate. Compare with the clean seeds. Run DFBScanner [5] and HTell [6] too.

Vary θ and sparsity. Record the attack gain over the clean model and the AUC of the best statistic.

Compare where detection fails with the sparse-PCA thresholds [9], using σ².

My prediction is that at high θ it is easy to tell the backdoored spectrum from the benign one, and that the published θ = 0.1 is already high. If the inserted layer stays close to its initialisation, θ = 0.1 is about say 150 times the weight variance, so I expect one eigenvalue far outside what the clean seeds show, and an AUC above say 0.90.

Going down in θ, I expect detection to hold as long as the backdoor still helps the attack. The trigger alone already fools the clean model most of the time, so the backdoor has little room to help at small θ.

I would drop the question in two cases:

At θ = 0.1 the clean seeds show eigenvalues as large as the backdoored ones, so the AUC stays near 0.5. Then their conjecture has its first evidence, and I would write that up.

I cannot reproduce their attack success and clean accuracy, so I am not testing their construction.

Given I did not find code of the original paper, the first month would be devoted to

reproducing the main paper results

make a baseline

ideally, I get in touch with someone really good at math to provide a proof or not of the paper’s conjecture

The second month, and first half of the the third month

start from the basic paper’s setup for theta = 0.1 and use spectral methods to detect the backdoor. Produce first results.

the sweep over parameters would produce space where conjecture is false

Rest of the time in writing notes by hand, no AI

I would need access to Runpod GPU, say 300-500 hours of an A100

We train small transformers on Mess3 with a backdoor i.e a fixed 4-token trigger that switches the tokens to a second HMM. We want to find out whether, on normal inputs where the trigger never finishes, the residual stream still tracks how much of the trigger has been seen, whether an attacker can remove this without making the backdoor less reliable, and whether the same check finds a planted tri

• •Often when checking a model for a backdoor we don’t know what the trigger is.
• •Before the trigger is finished, how much of it has appeared doesn't change the model's next prediction at all, so there’s no obvious reason for the model to keep track of it.
• •It does change what could happen a few tokens later though, and Shai et al. [1] found that transformers store information about the whole future, not just the next token. We aim to investigate whether normal training forces a backdoor to leave a trace, or whether a model can hide one the way the hand-built backdoors in [4] and [5] do.
• •If the trace is there and hiding it costs reliability i.e defenders get a way to look for backdoors without knowing the trigger, and attackers face a trade-off we can measure
• •If an attacker can hide it for free i.e learned backdoors can be as hidden as the hand-built ones, and probe-based detection like [2] can't be trusted against an attacker who knows about it
• •Known answer i.e in the toy process we can work out exactly what the model should be tracking, so we can check what a detector actu

• •Shai et al. [1] showed that transformers trained on data from a hidden Markov model (HMM) store their beliefs about the hidden state as a geometric shape in the residual stream, and that this shape holds information about the future beyond the next token. Their processes had no rare events though, so every part of the belief mattered for prediction all the time
• •Hubinger et al. [3] showed that backdoors trained into LLMs survive safety training
• •MacDiarmid et al. [2] then showed that simple linear probes on those models can tell when the backdoor is about to go off, using generic yes/no prompts that don't mention the trigger
• •But this is on inputs that contain the trigger, the attacker wasn't trying to dodge the probe, and we don't know what an LLM is really tracking inside, so it's unclear what the probe is picking up
• •Goldwasser et al. [4] and Draguns et al. [5] build backdoors by hand that are provably hard to detect, by hiding the trigger inside cryptography. This shows detection can fail by design, but says nothing about backdoors a model learns in training, or what hiding one costs
• •Piotrowski et al. [6] covers how one layer computes the belief update on Mess3, but only for the normal process with no backdoor
• •So no one has checked whether a learned backdoor shows up on normal inputs, whether an attacker can train it away, or whether a detector tested where we know the answer works on a real LM

• •Toy experiment on finding trace (Pt1)
• •We can build the backdoored process as one HMM. The hidden state is the normal Mess3 state plus how many trigger tokens have been seen in a row so far (0 to 3). Once the trigger is complete, the tokens come from a second, different Mess3 for the rest of the sequence
• •The code from [1] gives the exact belief for every input
• •Train small transformers with the setup from [1] + model trained on normal Mess3. First check the loss is close to the best possible, both overall and right after the trigger, so we know the backdoor was learned
• •Then, on normal inputs only:
• •Run a direct check at each layer, train a linear probe to read off trigger progress. Progress only depends on the last 3 tokens, so the control model shows how much you can read for free from recent tokens
• •Blind check (what a defender could do without knowing the trigger) i.e train a probe for the normal Mess3 belief only, then group what it can't explain and see if the groups match trigger progress
• •Plot the main directions of variation (PCA). Here in theory we predict a separate copy of the Mess3 shape for each progress value
• •Vary trigger length (3 to 6) and number of layers to see when the trace appears / disappears
• •Can an attacker hide it? Toy + theory (Pt 2)
• •Theoretically we want to prove two normal inputs that differ only in trigger progress lead to different futures, so any full record of the belief has to include progress (this part is close to the definition). How different those futures are is limited by how likely the trigger is to finish soon, seems like about 1 in $3^{L-k}$ for progress $k$.
• •Attacker 1 uses triggers with no step-by-step structure i.e a random set of 4-token strings with the same total rate. Same checks as Part 1
• •Attacker 2, training against the probe i.e we train a probe for trigger progress at the same time as the model, and penalise the model whenever the probe succeeds (gradient reversal, as in [7])
• •For each attacker, plot how big the trace is against how reliably the backdoor still works.
• •Fine-tune on normal Mess3, like the safety training in [3], and check whether the backdoor and the trace disapp

• •Toy model prediction: I believe trigger progress is much easier to read in the backdoored model than in the control, the PCA plot shows a separate copy of the Mess3 shape for each progress value + blind check matches progress well above chance
• •Attacker prediction: the trace gets fainter as the trigger gets rarer, roughly as we claim, and both attackers can shrink it but only by making the backdoor less reliable
• •Real model example: the blind check picks out the trigger well above chance against the plain backdoor, and does worse against the attacker
• •What would change my mind:
• •The backdoor works fine but trigger progress is no easier to read than in the control. Then the model only checks for the trigger at the moment it finishes, the belief geometry result doesn't cover things the next prediction doesn't need so the attacker/ real model ideas would fall through
• •An attacker removes the trace with no loss in reliability. This means the trace isn't forced, and learned backdoors can be as hidden as “hand built” ones
• •The check works on the toy but finds nothing in Pythia. Then the toy result doesn't carry over, main work here would be to find which of our assumptions failed/ fell through

• •Month 1: Part 1 with the 4-token trigger
• •Month 2: Part 2, both attackers, trace vs reliability plots
• •Month 3: Part 3 on Pythia-160M.
• •If it ran over the time, we can drop the trigger length sweeps first, then the fine-tuning test, then cut Part 3 to one trigger without the attacker.