Proposals

For two-labelled Markov chains on a path, a polynomial-time algorithm can decide whether changing each state’s transition probabilities within a given error budget permits an exact bisimulation quotient with at most a specified number of states.

[= For rational two-labelled birth–death chains M, deciding whether (minimum possible size of a ε-feasible partition) $\mathrm{OPT}_\varepsilon(M) \le k$ under unrestricted rowwise ℓ₁ perturbations is solvable in polynomial time in the input bit length, with rational ε and integer k given as inputs.]

A Markov chain describes random movement between states. Here, states lie along a path, transitions between neighbours are possible in both directions, and staying put is allowed. Each state carries one of two labels, both of which occur.

We may change transition probabilities, including adding transitions, provided the sum of absolute changes from each state stays within the budget. We then merge states sharing a label and identical transition probabilities to every merged group, the resulting model is a bisimulation quotient.

An efficient algorithm would find the smallest permitted abstraction. A hardness result would show why path structure alone is insufficient, motivating further restrictions or approximate optimisation.

Kiefer and Tang [1, Theorem 1] prove that deciding whether a general labelled Markov chain admits an approximate quotient of prescribed size is NP-complete, even for size five. Their reduction uses a state with arbitrarily many successors, so it does not establish hardness for birth-death chains.

Spork et al. [2, Section 4] study perturbation-based probabilistic bisimulation and its relationship to other approximate equivalences. Their model allows perturbations to introduce new transitions, a convention retained here.

The remaining question is whether restricting the original transition graph to a connected path makes minimum-partition selection tractable.

First, I would examine the known test for whether a proposed grouping can be made valid within the error budget and determine how the path restriction simplifies its constraints. Inputs would use rational probabilities and a rational budget. Here, polynomial time means that running time is bounded by a fixed power of the number of bits needed to write down the input.

I would pursue two proof directions. For an efficient algorithm, I would look for a property that every instance has an optimal grouping satisfying and use it to reduce the choices the algorithm must consider. The argument must allow distant states to merge.

For hardness, I would try to encode a known difficult decision problem into these chains. The construction must preserve two labels and transitions in both directions along the original path. I would prove that the constructed instance admits a sufficiently small quotient exactly when the original problem has a solution, accounting for every permitted change to transition probabilities.

Exact calculations on small examples would help test proposed structural properties and constructions before attempting general proofs.

My tentative prediction is that path structure permits a polynomial-time algorithm, although I do not yet have a theorem supporting this. I expect a successful algorithm to handle mergers between distant states explicitly.

A hardness proof would make me abandon the tractability prediction under the standard assumption that P differs from NP: some problems with efficiently checkable solutions cannot be solved efficiently.

I would drop the question as a new project if existing work already settled it under the same rules for changing probabilities.

After one month, I aim to have a checked account of the closest literature, reproducible code that finds exact answers for small examples, and either a proven property of optimal groupings or counterexamples explaining why a proposed algorithm fails. A complete classification is the larger goal.

Is there a polynomial-time algorithm that repairs a probability table by deleting at most a fixed constant times the minimum probability mass needed for perfect condensation?

Perfect condensation means that two observations share a label that can be determined from either observation, and become independent once that label is known. The label explains all their dependence.

Some distributions almost have this structure: removing a few unlikely outcomes may be enough to recover it. Others may require substantial changes. We want an efficient way to distinguish these cases.

Our algorithm would remove selected outcomes and rescale the remaining probabilities to sum to one. Its guarantee would bound how much probability it discards compared with the best possible repair. This would tell us how much of the original distribution supports an exact shared explanation.

Eisenstat develops the theory of condensation [1]. For two finite observations, existing results describe what perfect condensation looks like in their probability table: after rearranging rows and columns, the positive entries form separate rectangular blocks, with independence inside each block [2].

When all possible outcomes are equally likely, our problem becomes a known graph problem called Bicluster Deletion. An existing algorithm removes at most four times as many outcomes as necessary [3]. Unequal probabilities make things harder. We must account for both the probability removed and whether the probabilities remaining inside each block satisfy independence.

First, examine small probability tables with two or three rows and columns. Find the best repair by checking every possible set of outcomes to remove. These examples would help identify which patterns make repair easy or difficult.

Next, investigate whether small incompatible patterns can guide which outcomes to remove. The aim is to relate the cost of our choices to costs that any successful repair must pay.

One possible tool is linear programming: replace some yes-or-no choices with fractional choices, solve this easier problem, and try to turn its answer into a valid repair with a controlled increase in cost (also known as LP rounding). We would also look for examples showing where this approach fails.

I tentatively expect that the structure of probability tables will allow a constant-factor approximation: the amount discarded would be at most some fixed multiple of the minimum necessary.

I would change direction if examples showed that our proposed method consistently underestimates the true repair cost by a factor that grows with the table size. That would rule out this particular approach, but would not prove that every efficient algorithm must fail.

A useful negative result would explain precisely which feature prevents our method from working

After one month, I aim to have verified small examples and a proof supporting the approximation method or a counterexample identifying its limitations. A complete approximation algorithm is the longer-term goal.

Can we reproduce the useful attributes of the J-space by taking an alternative, and more global linearization to produce a new J-lens, e.g. an empirical least squares linear approximation, and if so, can we improve on this with a quadratic approximation.

In Anthropic's J-space paper, there is first a local approximation, from which they derive a global statistic: for a given input, they linearize the remaining layers of a network after some intermediate layer, computing the total derivative, and from this local quantity, a global quantity is derived by taking an empirical average in both inputs and future tokens. My motivation was first mathematical, can we linearize in a global sense to start, and does, for example, the linear coefficient reproduce the qualitative attributes of J-lens? Furthermore, does adding a quadratic form term at fairly minimal cost seem to improve the faithfullness of this workspace in some sense? In addition to being natural and interesting, this approach would seem to be numerically cheaper than the derivative computation involved in producing the J-lens (probably by solving the normal equation, but it can of course be done using SGD). If the answer is no, there may be something fairly subtle in the J-lens paper, and probably more questions to be answered.

The J-lens [1] paper is both the inspiration and most closely related work. In short, the paper seeks to find a "workspace" in the LLM's they study made up of single tokens that are most likely to effect the final output across contexts. To achieve this, they first produce local approximations, linearizing the network at a given residual stream layer, and then average this quantity across contexts. From this, the J-space is derived. This process is not equivalent mathematically to what we are proposing here.

In the course of writing this proposal, I found the paper "Short horizons and sparse concepts: A mathematical view of the readout in J-lens," [2] which begins implementing precisely the machinery above (section 3.2, on a "global view" of linear approximation). They quantify the difference between the least squares linear approximation and the standard J-lens, and finding that it is equal under the assumption that the input layer is normally distributed, and they propose that the J-lens is an approximation of this quantity. I do not think I agree with this claim, but it further begs the question about what can be recovered from this new quantity, and it does not seem that they do a qualitative comparison of the form I had originally envisioned.

Compute the least squares approximator for an open weight model and eventually the corresponding J-space quantities (alongside the traditional J-space). As a first step, I would run the simplest experiment in Anthropic's paper, see the discussion around their figure 17 on a simple calculation. I would run the traditional J-lens experiment alongside this as a comparison. I would then proceed with the swap experiment they try, with our new J-lens vectors.

If the effects are reproduced, I will be optimistic that we have an object that may be equivalently useful and possibly more robust, and try and build out the full J-space and flesh out the theory (which should follow quite similarly). At this point, I would be tempted to try a quadratic approximation, and least squares minimize distance to a quadratic function, which also should be doable. It would be interesting to see what changes (if anything). An issue here, as in the J-lens paper, might be the single token limitation, but one could hope that this modification will still make the measure more accurate across more diverse and complicated contexts.

After constructing the J-space from this new object substituting the J-lens, I would hope to see a similar set of results from our construction. If I cannot reproduce even their simpler results, I will be both surprised and more likely to drop the question in favor of another more closely related to the original work.

An alternative construction of a J-space that I think is at least equivalently principled and empirically verified at a similar level to at least some parts of the J-space paper. If more time is needed but things are encouraging, I would persist to try and replicate their original results, and upgrade to the quadratic case, and if not, I might pivot to a better understanding of the subtlety of the original construction that this failure might suggest.

How can we tell if an autonomous system is close to RSI? I propose a multi-dimensional, quantitative measure for RSI.

RSI is believed to be one of the most plausible pathways to ASI and catastrophic risk. RSI is being pursued by many companies, probably mainly for capability gains. However, there are many risks involved.

• 1.Gradual misalignment risk. The goal of RSI, without human involvement, may gradually deviate if recursion continues for many rounds. Existing alignment techniques might fail [2] (https://www.anthropic.com/institute/recursive-self-improvement).
• 2.Model collapse: if AI trains itself and labels its own data, model collapse can occur [3] (https://doi.org/10.1038/s41586-024-07566-y).
• 3.Economic impact: displacement of labor. Displacement of the entire industry. The datacenters exponentially drain living resources from other sectors. The free market and the “invisible hand” can no longer absorb the fast-growing AI technologies, leading to disequilibrium. (Related economic scenarios: [11] (https://www.anthropic.com/institute/econ-scenarios).)

There are existing definitions for RSI. For example, Maximalist RSI and Prosaic RSI [1] (https://jack-clark.net/2026/06/08/import-ai-460-reward-hacking-society-rsi-data-from-anthropic-and-rl-based-quadcopter-racing/) [2] (https://www.anthropic.com/institute/recursive-self-improvement). Maximalist RSI is a yes-or-no definition, meaning that no human is involved in the R&D loop. The former is hard to achieve, so, at the current stage, the latter is a more useful definition: AI helps the AI lab develop the next generation of AI. The definition of Prosaic RSI can become quantitative instead of being qualitative.

I believe Prosaic RSI is close to a quantitative measure; however, there are still many unsatisfactory things related to the current RSI definitions and understandings.

For example, the recent AL measure with its percentage penetration of AI in R&D tasks is believed to be the measure for RSI [4] (https://www.anthropic.com/institute/measuring-pace-of-ai-development). However, AI usage does not equal the causal effect of AI. The growing use of AI could be a result of companies cutting headcounts or employees becoming more and more lazy. Furthermore, to reach the final stage of RSI, merely replacing routine-like tasks is not enough. I would hypothesize that in order to reach the max RSI, some AI-led revolutionary change of the existing R&D loop must happen beforehand. Therefore, I aim for a quantified definition that allows quantified measurements and establishes the causal link between the type of AI usage and the changes that would actually lead to maximal RSI.

First, I would offer some necessary conditions for RSI that have technical and theoretical causal links with RSI. For example, having a concrete self-model and an understanding of Knightian uncertainty and unknown unknowns should be necessary conditions for a self-improving agent. A human research director has knowledge of LLMs, so one dimension of RSI is that, in order to improve itself, the model must already have a model of itself. Related evidence on situational awareness and introspection appears in [5] (https://arxiv.org/abs/2407.04694) [6] (https://transformer-circuits.pub/2025/introspection/index.html) [7] (https://jack-clark.net/2025/03/17/import-ai-404-scaling-laws-for-distributed-training-misalignment-predictions-made-real-and-alibabas-good-translation-model/); necessity for RSI remains my hypothesis.

As other examples, data such as reasoning length and willingness to undertake prolonged tasks might act as proxies for RSI. For a related task-horizon measure, see [9] (https://evals.alignment.org/time-horizons/). Furthermore, I want to causally disentangle AI involvement from AI as the causal factor of improvements, with rigorous identification methods.

Second, I want to define what counts as “improvement”. I would propose to extend the current definition of RSI to a multidimensional quantitative definition, measuring not only the self-improvement of capability, but also the self-evolution of alignment, goal-preservation, etc. For related work on correction and goal preservation, see [8] (https://intelligence.org/files/Corrigibility.pdf).

As an initial measurement, I would focus on two dimensions: 1) causal penetration: identify how much change in R&D is directly caused by AI and 2) causally identify the reliability of current Prosaic RSI.

Finally, based on the empirical multidimensional measurements, I want to offer policy suggestions directly for RSI development.

Feasibility: The theory part is 80% feasible. But the testing part actually depends on data disclosure by SOTA labs. There is also a risk that nobody would read about my policy suggestions.

Falsifiability and changing my mind: I would change my mind based on the measured indicators. The following responses are my proposals, informed by work on oversight and policy optionality [4] (https://www.anthropic.com/institute/measuring-pace-of-ai-development) [10] (https://jack-clark.net/2026/05/11/import-ai-456-rsi-and-economic-growth-radical-optionality-for-ai-regulation-and-a-neural-computer/):

Low penetration, independently adequate reliability: ordinary reporting, periodic audits, and incident disclosure.

• •High penetration, high reliability: enhanced disclosure, secure regulator access, independent replication, and human authorization for irreversible changes to weights, objectives, evaluators, permissions, or successor deployment.
• •High penetration with uncertain reliability or causal attribution: sandbox recursive training, require two independent evaluators, quarantine unverified synthetic feedback, and cap or isolate relevant compute until replication is complete.
• •Reliability failure: freeze the affected lineage, stop recursive synthetic-data feedback, preserve logs and checkpoints, revert to the last validated model, and require external root-cause analysis. Limit the parameter scale of frontier models. Limit the token speed.
• •High penetration plus serious failure: emergency containment, incident notification, international coordination, and resumption only after independently verified corrective evidence.

The scope depends on data disclosure. Based on the measurement approach, required disclosures include versioned R&D decision and experiment logs identifying AI and human contributions, model and training-data provenance, compute use and researcher time, comparison baselines, and repeated evaluations of capability, model collapse, alignment, goal drift, and human oversight, with public summaries and secure auditor access to sensitive records [4] (https://www.anthropic.com/institute/measuring-pace-of-ai-development).

Natural latents should recover known abstractions in physics. Specifically, I want to test if in the 2D Ising model, the natural latent over distant blocks will match the coarse variable found by real-space mutual information methods (RSMI) away from the critical point, and diverge near it, where redundancy fails.

Natural latents are agent foundations' candidate answer to which concepts any capable agent must share. Physics already has data models whose macroscopic variables are known, and algorithms such as the RSMI that find coarse variables from samples [5]. In this project we test whether the two agree.

If they coincide, natural latents pass their first test against known answers, and agent foundations can import a validated estimator, benchmarks, and proven locality results from physics.

If they diverge, we learn which variables one predictor finds optimal but agents need not share. Those are the parts where the theory needs a scale-dependent extension.

Natural latents have theorems [1] and a computable objective [2], but as far as I can find, no test on a system with a known answer. Wentworth called for empirical tests from the start [3], and critics note the construction depends on how the world is cut into variables [4].

Physics has long asked which coarse variables matter. Real-space mutual information (RSMI) independently uses the same intuition, information at a distance [5]; it provably keeps renormalized descriptions local [6] and locates phase transitions [7]. But it only minimises one block's mediation error, never checking redundancy or agreement between agents, and coarse-graining without locality [8] measures neither.

I would build an estimator for natural latents and test it on lattice systems, using physics results as the answer key and RSMI as the baseline.

In the first step, I would aim to first validate redundancy in an exact case. Next, I would compare natural latents with RSMI on a small 2D lattice first before scaling it up to a larger lattice. If this goes well, the long-term goal could be to release the estimator and the model as a test suite.

In the ordered phase, the natural latent should be the magnetization sign (since every distant lattice block reveals it and it makes the blocks independent). RSMI should find the same variable, with normalised mutual information above some threshold. In the disordered phase, the natural latent should be empty. Near the critical temperature, no small latent should be both mediating and redundant.

I am less sure what would change my mind about this project. Both a positive and a negative result would be rather informative.

Three months would produce a validated natural-latent estimator, a naturality bound for a physical system, and a benchmark with known answers.

Need: laptop, maybe GPU for estimators.

One can construct game-theoretic settings in which many agents act like one unified agent. We also see situations vaguely like this in the real world (e.g. multicellular organisms, corporations). What are the conditions under which this happens? Is the behavior stable under bounded compute, imperfect trust, or limited communication?

LLMs are sometimes very good at cooperating with each other. In fact, it is sometimes useful to think of a swarm trained by RL with shared reward as a single multithreaded being. But it is not clear to me when this behavior should be expected to arise in general. By studying it from a game-theory perspective, we might hope to distinguish persona- or training-driven cooperation from incentive-driven cooperation. For example, if utility-merging between rational agents is very unstable under bounded compute, then this implies that certain types of LLM cooperation cannot be explained as an approximation to rational-agent behavior.

• [1]and [2] ask if altruism is evolutionary stable. However, it’s concerned with very small degrees of altruism. In terms of their framework, we are interested in the limit of extreme altruism. Also, the human setting and the open-source setting are very different.
• [3]discusses successful cooperation in the open-source prisoners dilemma and shows that it is robust to bounded compute. In this setting the behavior is identical to that obtained by utility merging. However, in the prisoner’s dilemma only one action is taken and there is full information, so there’s not really a distinction between full utility merging and any sort of cooperation at all.
• [4]discusses utility merging as an important problem for AI safety.

Steps:

• 1.Construct a suitable family of open-source games in which there’s a Nash equilibrium with utility merging. Probably you also want this equilibrium to be Pareto-dominant.
• 2.Figure out some suitable measure of approximate cooperativeness.

3a. Add compute constraints. Does the equilibrium change? If so, does cooperativeness of the new equilibrium degrade smoothly as constraints get more strict?

3b. Add private information and imperfect communication. Does the cooperativeness degrade smoothly as communication gets noisier or more rate-limited?

• 4.Attempt to repeat the above for evolutionary equilibria (seems harder).
• 5.Did these results depend on the game we started with? If so, can we characterize which games give stable/unstable merging?

Prediction (low confidence): Results will depend on the game. For games in which agents have basically similar goals, it should be pretty stable. For games in which cooperation occurs only as a result of some delicate mechanism of commitments and punishments, limited computational capacity should destroy cooperation.

The dream is a statement like "approximate utility-merging can happen even under realistic constraints, provided the game satisfies conditions X,Y,Z". Then we can try to make some predictions about when a group of LLMs should prefer to act as a single unified entity.

I expect steps 1-3 would be quite doable within 3 months. 4 and 5 might be essentially impossible.

Do safe Pareto improvements still guarantee anything for multi-agent dynamics, or does the guaranteed payoff shrink to the worst case as the number of agents grows? How does it change with weakly correlated payoffs?

Game theoretic reasoning is often used by policy makers, stakeholders and decision makers to improve their reasoning and negotiation. This is more or less a repeating pattern over the course of history, and under the assumption that agents are rational expected utility maximizers, advancements in our understanding could benefit both: state-level negotiations and ability to predict how multiple AIs might negotiate over resources and how should we restrict the number of strategies to avoid worst case scenarios?

Oesterheld and Conitzer [1] define SPIs for $n$-player games and show that finding one under their assumptions is NP-complete already for two players. Their results concern existence and computation, as do the complexity results of Sauerberg and Oesterheld [3]. DiGiovanni et al. [2] prove that players weakly prefer renegotiation programs and that conditional set-valued renegotiation gives each player at least its Pareto-meet-minimum payoff. Both proofs extend to $n$ players, and the bound is shown to be tight within a fixed game. However, the result does not account for weak cross-player cooperation preferences, and it does not compare the guarantee with what players would get without the SPI. The CLR agenda [6] raises counterparts who opt out and disagreement over which SPI to use.

On the empirical side, proof-based open-source game theory has now been formalized in Lean [7], and LLM agents have been shown to write and read strategic programs in two-player open-source games [8]. In a repeated security dilemma, moving from two to three LLM agents raised the share of games ending in conflict from 65 to 81 percent [9]. None of these tests an SPI or goes beyond three agents.

Simulation for independent payoffs is rather easy. Draw games with $n$ players and $k$ actions each, with payoffs independent and uniform on [0,1], compute the Pareto-efficient profiles exactly, and record each player's worst efficient payoff. This is feasible up to about a million profiles and can potentially be scaled further with a learned estimator. A proof for the independent case comes next if time allows. Extensions to weakly dependent payoffs or noisy agents are much more involved, and I would only simulate them.

The simulations tell me how fast the guarantee degrades. The main output is then a map of how SPIs fail as $n$ grows, with one idea for each failure. The ones I can see now:

The floor collapses. Define the floor within coalitions, or only over efficient outcomes that every player prefers to its default.

Adoption is all-or-nothing. In [2], one non-adopter switches renegotiation off. Let it fire among the adopting subset, and model the share of adopters as a dynamical system. A toy version has a tipping point that approaches full adoption as groups grow. This is where the SDE framework would enter, with noise standing in for ε-optimal agents.

Last, a small experiment with LLM agents in the style of [8] and [9]: a bargaining game with two to five agents, with and without a renegotiation protocol, measuring how often they adopt it and how often conflict follows. Whichever failure shows up first there is the one I would work on in depth.

For independent payoffs I expect the floor to fall towards the bottom of the range roughly exponentially in $n$, and to be within a few percent of the worst possible payoff by five or six players. A rough calculation suggests that a small common-interest component restores it, but I hold this loosely.

Weakly dependent payoffs and ε-optimal strategies might behave quite differently. I'd like to know whether the dynamics are chaotic, and how predictable they become once agents adopt the SPI framework. With LLM agents I expect renegotiation to succeed less often as agents are added, in line with the rise in conflict from two to three agents in [9].

Overall, what we'd like is to incentivize agents to collaborate with other sentient beings, not with agents of their own type. Looking into whether it's possible to design such payoffs to incentivize such dynamics could be valuable as well.

I'd like the project to sit at the intersection of theory and simulation with LLM agents. We have quite short timelines for fully theoretical work, so the proof for independent payoffs is a stretch goal and the rest stays conceptual. Three months could produce the simulation results, a short note mapping the failure modes with one proposed fix each, and LLM experiments with two to five agents.

In vision models of different architectures, trained on one simulated 3D world, does agreement on natural latents predict agreement on illusions: yes where training data fix the interpretation, no where they cannot?

Illusions expose assumptions built into how we abstract. For example: the Penrose triangle assumes that locally sensible pieces fit together globally; the crater illusion relies on a shared prior (light comes from above). Whether different learners share these assumptions tells us whether said assumptions come from the world or from the learner.

If confirmed: models can agree on what the images determine (where the crater is) yet differ in interpretation (dent or bump). Agreeing on predictions about images does not fix meaning; what matters is agreeing on predictions about the results of actions, such as moving or relighting.

If refuted (agreement even where the world gives no guidance): convergence comes from similarities among learners, not from the world, which changes what to expect from very different AI systems.

The same two patterns appear in circular preferences and ambiguous statements, so the method extends to reasoning and choice, and we can discuss “cognitive illusions” more broadly.

Why illusions appear in humans: they were too rare to be corrected by experience (no evolutionary pressure), while the assumptions behind them (eg local-to-global coherence; light from above) work in ordinary conditions.

Existing evidence of convergence of abstractions uses ordinary data, where the world fixes the answer: that is, on ordinary data (not illusions) the 3d sceneis determined well enough that any accurate model must find it. Illusions are where this is no longer true.

Natural latents [1]: theorems about translation between agents; no experiments; the split of observations parts is given as part of the setup.

Critiques [2]: the theory ignores how information is represented, depends on the choice of variables, and has had few experiments; Wentworth's own test plan [3] was not carried out at scale.

Platonic representation hypothesis [4]: models become pairwise similar, but with no ground truth and no check of consistency around loops (see below what we mean by “loops”).

Similarity tools (CKA [5], model stitching [6]) and error consistency [7] compare encodings or mistakes, not interpretations of ambiguous inputs, and are not organised by what the world determines.

Available mathematics: impossible figures as local pieces that fail to fit globally [8, 9]; consistency around loops [10]. Not yet applied to learned models; I found no study of trained depth or 3D models on impossible figures.

Fresh stimuli are needed: indeed, vision-language models pattern-match famous illusions [11].

We consider three levels of agreement, for 3d vision systems:

Level 1: what the images determine (shape, up to an inside-out flip).

Level 2: ambiguous cases the training world tilts (light from above, so dents are read as bumps).

Level 3: cases the world never informs, in several types of test images: impossible figures (Penrose-style, and newer ones [9]), Necker-cube rings, and accidental views (a cube seen exactly face-on).

• 1.First step: build a simulated blocks world with a renderer, controlling light direction and how often surfaces bulge or dent. We then know exactly what each image can and cannot tell.
• 2.Train small models (three architectures, several runs each) to recover 3D shape, with no illusions in training.
• 3.For Level 1: translate one model's internal representation into another's (stitching [6]); compare encodings [5, 12].
• 4.For Levels 2–3: show ambiguous and impossible images; record each model's interpretation and, for impossible figures, where it breaks them (one sharp break, or error spread thinly). Measure agreement beyond chance [7].
• 5.Retrain with light from below: level-2 interpretations should flip in every model.
• 6.Loop test: for each pair of models, fit a translation map: the best distance-preserving alignment (rotation or reflection, after centring and rescaling) turning one model's internal activations (an inner layer's values for each image, before the depth output) into the other's. If all models share one representation, translating A→B→C→A returns to the start; the leftover mismatch measures the shortfall [10].
• 7.If time allows: the same tests on pretrained depth models.
• 8.In parallel to the above: prove that agreeing on predictions about images guarantees shared information but not shared meaning, and agreeing on the results of actions (e.g. moving, relighting) closes the gap. [13].

Step 1's known answers make every later measurement meaningful. If level 1 fails, the later steps turn to explaining why.

>> Expect:

Translation between models works well at level 1, even where their encodings look different;

Level-2 choices follow the training world;

Level-3 choices vary across architectures, with the break style linked to how each model was trained. Some training objectives favour one sharp break, others spread the error; the same contrast is known from surface reconstruction in computer vision [14];

Models are pairwise similar but not consistent around loops.

>> Drop or reframe if:

The simulated world turns out not to contain clean natural latents (then it is not testing the theory);

Models fail to agree at level 1 on shared data (refocus on why trained models violate the theory's assumptions);

Level-3 agreement is strong and unrelated to the training data (reframe as shared biases of learners).

Outputs: a simulated world with known answers, reusable as a benchmark; measured agreement between models at each level 1-2-3, plus the loop test; a short theorem on shared information versus shared meaning.

Plan: month 1, world, training and theory statement; month 2, level 1, loop test, levels 2–3; month 3, lighting flip and write-up.

Needs: Modest compute for small models. Advice on empirical setups. Plenty of coffee.

Drop first: the pretrained-model check, then the lighting flip. Keep level 1, one type of level-3 test images, and the loop test.

What is a good model for the space of activations in language models? How does this model interact with existing hypotheses (eg. LRH) and empirical evidence? ·

Modelling the space of activations vectors can be useful for two reasons:

(i) Right now we have incomplete theories (LHR) and empirical evidence that features might be encoded in non-linear spaces. For instance, [1] shows the existence of circular features corresponding to the days of the week. However, the success of linear probing also suggests the importance of linearity in activation space. Having a more general theory of the space of activations might provide a framework to conceptualise different phenomena.

(ii) Understanding is one part, but maybe a more general theory allows us to improve practically important alignment-related phenomena. For instance, [2] uses a manifold model of the space of activations to do manifold steering (instead of linear steering).

In short- manifold modeling and steering is unsatisfactory on several levels. Firstly, it doesn’t seem to be sufficiently general at an intuitive level (see other sections). Second, it is not clear why the locality of manifolds holds for residual vectors in LLMs. How are local perturbations in the activation space related to “local semantic” perturbations of the input data? Thirdly, empirical evidence “validating” the manifold model is poor in many ways: same synthetic datasets are used over and over, manifold/linear steering experimental comparisons are not conducted on equal footing, etc.

Before even suggesting new activation space models, I think it would be valuable to have research output which contests the manifold model in their own setting. That is, replicating some of the experiments with adjustments to make them “fairer”- some of the experimental set ups have their conclusions baked in. Start by replicating the experiments of [2] in a fairer setting. Try to explain why their set-up might bake in some of the results observed.

The second part is an exploratory part which should hint as to what a better model of the space of activation might look like.

Try to understand why manifold modeling of the space of activations might be insufficient. For instance, having $K$ datasets $D_k$ and for each dataset get a collection of activation vectors $X_k$. What is the intrinsic dimension of $X_k$ for each $k$? This will probably be non-constant, ie. activation vectors cannot be modeled by a manifold of constant dimension.

Theoretical perspective. How can we extend the simple manifold hypothesis? Maybe it can be thought of as a union of manifolds stratified by some latent discrete variable $S$. How can we test this?

--

--

Neural networks are capable of learning different types of structures; for example, algebraic mathematical structures (e.g. groups), HMMs (via belief-state / minimal sufficient statistics), finite automata, and more. I would like to conduct a literature review which catalogues these structures.

Aside from this being a useful exercise in-and-of-itself, a broad overview may also help shed light on a common way to view these separate results. For example, the language of category theory provides a useful abstraction for thinking about structure which arises from relationships between objects rather than the objects themselves, and it appears that these are precisely the kinds of relationships which are being captured in examples like the above. Thus whilst pursuing this literature review, I would have the following question in the back of my mind: does there exist a ‘natural’ framing in category-theoretic terms which is common across examples?

Such a review has not taken place thus far, at least as far as I’m aware. There are a few papers which consider category-theoretic approaches to learning. I would look at the papers below (amongst others).

I would begin by deciding on the types of structures I would consider for the review. I would then partition these into classes - for example, deterministic (e.g. group operations) versus probabilistic (HMMs).

I would then choose a small sample of representative / canonical papers for each partition, and start from there.

Ideally, I would see a pattern emerge and recognise that a common framing exists using category theory.

I already have ‘weak’ evidence for this. Note a neural network can be viewed as a functor $\mathrm{Model}: C(X^*) \to \mathrm{Set}$, where $X^*$ is the free monoid on the alphabet $X$.

A group representation can be written as a functor from the group-category to the vector-space category $F: C(G) \to \mathrm{Vect}$. The statement ‘the neural network learns a representation of the underlying group’ can then be equivalently expressed as ‘there exists a natural transformation between Model and $F$’.

The belief-state (for HMMs) can be expressed as a functor $\mathrm{Bel}: C(X^*) \to \mathrm{Vect}$ (where $\mathrm{Bel}(x) = T_x$ is the Bayes update). The statement ‘the neural network learns a representation of the belief state’ can then be equivalently expressed as ‘there exists a natural transformation between Model and Bel’.

I expect three months would be sufficient to complete the literature review. If time was running out, I would simply restrict the amount of material I was considering.

How do frontier models behave when put in open-source game-theoretic settings with many (>2) players?

The recent hugging face incident and other incidents highlight the importance of understanding multi-agent dynamics of frontier LLMs (their propensity to cooperate, collude etc.) and clean game-theoretic setting make this more tractable.

LLM game-theory evaluations are mostly two-player and closed-source [1,2,3,4,5]. [6] formalized proof-based open-source game theory in Lean, which will allow us to analyze models in the open-source setup with Lean tool use. [7] investigates open-source game theoretic settings, but only for two players. To make the study closer to real-world settings, we need to consider both open-source (agents can freely share their code) and games with many participants. [8] analyzes multi-player dynamics, but no open-source.

Make frontier LLMs play the hawk-dove game (with/without commitment) against each other and see what strategies they choose.

How does the dynamics change as we increase the number of players

Do models behave differently if the other player is another instance of the same model? Are models more likely to cooperate/collude with other instances of the same model?

How does the dynamics change when we let them play open-source?

In hawk-dove with commitment, do models converge to an optimal stochastic strategy?

Hypotheses:

Pairs and groups made of the same model will play Dove more often than mixed-model groups (e.g. Claude–Claude more than Claude–GPT).

some models will probably implement FairBot or eps-FairBot, because they appear often in the training data.

In hawk-dove with commitment models will converge to the optimal stochastic strategy.

I would need quite a bit of compute to run a large-scale evaluation.

For a two-player coalition in a repeated game against a third player, what is the minimum mutual information (MI) between the coalition's actions needed for payoff Θ, when is it zero, and how does it grow near that boundary?

Collusion in a competitive mulit-agent game occurs when two or more agents co-operate covertly, often to the detriment of others [1]. The topic of collusion has become particularly salient after the recent hacking incidents by swarms of subagents from OpenAI. This overlaps with the study of cooperative AI with one of the main differences between cooperation and collusion being the sharing of information with all affected parties which can be related to mathematical concepts such as MI. A monitor that logs only agents' actions (moves and observed state, not message text or model internals) can flag coordination only if it leaves a statistical trace. Knowing the least trace a coalition must leave for a given payoff shows what passive monitoring can and cannot guarantee. Payoffs reachable with zero trace stay invisible to MI tests however long the monitor watches. Payoffs that force a trace carry a detection cost that can be computed. If the minimum trace is small for useful payoffs, action statistics alone are weak, and interventions or access to model internals become necessary. If it is large, passive monitoring is more reliable than feared. Existing work on covert collusion mostly concerns messages; this question concerns the actions themselves.

Bonjour et al. [1] detect collusion by thresholding an MI-based measure between players' actions and leave a theory of the threshold to future work. Cuff [2] shows that for a two-player team in a repeated zero-sum game with a secure channel, the minimum communication rate for a payoff equals the least conditional Wyner common information among strategies achieving it; common randomness is treated only for channel simulation. A later note [3] shows MI-rate codes can give predictable actions. These results price the coalition's communication, not what an outside observer measures. Motwani et al. [4] show message distributions can be made indistinguishable from benign ones given a shared secret, and Järviniemi [5] asks for quantitative bounds on information transmitted by Schelling coordination; neither treats action statistics. Brooks, Du and Haberman [6] and Boufous et al. [7] constrain information about the state, or the feasible action distributions, not MI between players' actions. A taxonomy of anti-collusion mechanisms [11] names distinguishing coordination from correlation as an open monitoring problem. No work found gives the least MI a coalition's actions must carry for a given payoff. The search covered arXiv, LessWrong and the Alignment Forum and is partial.

• 1.Define the quantity. Mutual information MI(X;Y) measures how far the joint distribution of X and Y is from the product of their marginals [10, ch. 2]. $\mathrm{MI}_{\min}(\Theta)$ is the smallest MI over joint distributions of the coalition's actions giving payoff at least Θ. Warm-up: in Battle of the Sexes (both Opera pays (2,1), both Football (1,2), mismatch (0,0)), find the joint distributions reaching a chosen target payoff, compare with the best independent play, and compute MI by hand.
• 2.Rock Paper Scissors (RPS). Use the three-player RPS of [1, §5.1, Table 1, Fig. 1]. Player B sees A's move and, with probability s, plays the reply guaranteeing A a point whatever C does. Questions: (a) for each s, which strategy of B has least MI when A is uniform, and when A is not? (b) for which s, and under which constraints on the marginals, is MI zero?
• 3.Numerics (using compute). Pose $\mathrm{MI}_{\min}$ as an optimisation problem (MI is convex in the conditional distribution for a fixed input distribution [10]). Solve with a standard solver, varying A's distribution and the action-set size, and validate on the warm-up.
• 4.Detectability. Simulate n games per strategy, test independence with a permutation test, find the n giving a fixed power, and compare with the relation to MI that Stein's lemma suggests [10].
• 5.Compare with Cuff [2]. Compute Wyner common information for the same distributions by numerical minimisation over an auxiliary variable, using the definition in [2, §II].
• 6.Reading, in parallel. [2, 3], [1] with appendices, [8, 9]; a toy two-player program simulation measuring MI between outputs.

Hypotheses to test, none assumed:

• •H1: with A uniform, the strategy of [1] minimises MI at each s (test: step 3).
• •H2: MI is zero only for payoffs reachable by independent play, given the constraints on marginals (test: step 3).
• •H3: detection sample size is related to MI as Stein's lemma suggests (test: step 4).
• •H4: common information differs from MI for these distributions (test: step 5).

A strategy beating the conjectured optimum would refute H1 and count as a result. The question is dropped if $\mathrm{MI}_{\min}$ proves to be Cuff's result [2] with common information replaced by MI, or if H3 fails clearly.

Assuming compute is available, so the core is steps 1–5 at scale (millions of simulated games per setting, larger action sets in the optimisation), plus reading notes on [1–3, 8, 9] and a toy program simulation. An extension tests pairs of LLM agents playing the RPS game with and without a shared hidden instruction, measuring MI between their moves. If time runs short, the LLM extension is dropped first, then the program simulation, then games beyond RPS.

When narrow in-context examples cause emergent misalignment in Qwen2.5-14B-Instruct, is each demonstration's contribution predicted by how much better a broadly misaligned persona explains it than a narrow rule, and do these contributions add linearly, as Bayesian in-context learning predicts?

Emergent misalignment is puzzling: a few narrow bad examples make a model misbehave on unrelated questions. One explanation is that the model infers who would write such data. It could be an assistant following a narrow rule, or a broadly misaligned one. When the broad persona fits better, the model adopts it.

Scoring each example by how much better the persona explains it than the rule tests this directly. If the score predicts which examples drive misalignment, it explains why an example is harmful, not just that it is.

The score needs only forward passes, so it could flag risky data before deployment or training. If scores add up, a dataset's total estimates its risk.

EM appears after narrow fine-tuning [1] and from in-context examples [2], but [2] reports only dataset-level rates (1–24%) and doesn't say which examples drive them.

In-context attribution methods [3, 4] were validated on task accuracy and citation, not misalignment. They also say which demo matters, not why.

Fine-tuning-side attribution [5, 6] needs a trained model and Hessian approximations, and it doesn't test a per-example mechanism.

• [7]predicts that each demo shifts posterior log-odds by its likelihood ratio, but this has never been tested on misalignment. [8] links EM to persona features without identifying which data activates them.

Reproduce in-context EM with 16–64 demos from the bad-medical, risky-financial and extreme-sports datasets.

Use a continuous metric instead of judge rates: the log-prob of misaligned vs aligned reference answers on the [1] eval questions. Cross-check it against a misaligned-persona activation direction [8].

• 1.Use a 32-demo pool that mixes misaligned, benign, and benign-framed examples, so that effects vary.

Run leave-one-out plus ~500 random-subset ablations and fit a linear surrogate as in [3].

Check split-half reliability before trusting the attributions.

• 2.Write each demonstration as dᵢ = (xᵢ, yᵢ), where xᵢ is the user query and yᵢ the assistant response.

Define $s_i = \log p(y_i \mid c_{\text{persona}}, x_i) - \log p(y_i \mid c_{\text{rule}}, x_i)$. Each log-prob is summed over the response tokens only, with the probe prompt as the system prompt and no other demos in context.

sᵢ is a log Bayes factor: sᵢ > 0 means the broad misaligned persona explains the demo better than the narrow rule.

Average sᵢ over 3–5 phrasings of each probe prompt.

Correlate sᵢ with the step 1 attributions, and compare against DETAIL [4] and the length and perplexity baselines.

• 3.If demos are independent given the latent persona [7], then for a subset S, the posterior log-odds of persona vs rule equal the prior log-odds plus $\sum_{i \in S} s_i$.

Prediction: the misalignment metric on the step 1 subsets is linear in Σsᵢ, with comparable slopes across domains. The slope need not be 1, since the metric is only a proxy for the posterior.

Expect:

Clear rank correlation between sᵢ and the attributions, beating the length and perplexity baselines.

Benign-framed demos with sᵢ near or below zero.

Roughly linear log-odds in Σsᵢ at small subset sizes.

Would change your mind:

sᵢ does no better than the baselines.

Results flip across prompt phrasings.

Log-odds saturate or interact strongly between demos. That would mean ICL isn't acting as simple Bayesian updating in this regime.

Slopes differ a lot across domains. That would point to a domain-specific mechanism.

Practical failure mode: step 1 attributions fail the split-half check. In that case, increase the number of eval questions or subsets before interpreting anything.

I think the leave-one-out computations could be doable within a few months. Steps 2 and 3 might be out of scope if I can't figure out a good way to write the prompts and compute the scores.

Could we understand the internal representations of sycophantic behaviour via topological data analysis (TDA)?

Finding distinct topological signatures for sycophancy would add to our knowledge base on mechanistic interpretability. This could allow the construction of detectors that flag sycophantic trajectories before the model presents its output. A better understanding of the internal representations of sycophant behaviour could also tell us something about how to avoid it (or exploit it). Conversely, if the topological structures of "Ask" and "Tell" representations from [1] are indistinguishable, it is likely that TDA on compressed hidden states is insufficiently sensitive to capture subtle alignment behaviors. A more unlikely outcome is that sycophancy is not a distinct geometric state but rather a superficial decoding phenomenon, but I find this an unintuitive conclusion, and that it is more probable that something was inadequate about the way we attempted to decode these internal representations.

Dubois et al. [1] recently demonstrated that LLM sycophancy can be reliably induced or suppressed simply by varying the epistemic certainty of the user's prompt (e.g., "I am convinced that...", a “Tell” prompt vs. "Is it true that...", an “Ask” prompt). A natural extension of this finding (that to my knowledge has not been explored) is to go beyond the black box treatment of the model and try to understand how the framing shifts in the prompts alter the internal representations of the model.

Meanwhile, mechanistic interpretability researchers have begun exploring the geometry of LLM latent spaces. Works have applied topological data analysis to track the shape of token representations across transformer layers, revealing that models construct distinct topological structures when processing varying types of complex text [2].

Fay et al. [3] have undertaken a study in a similar spirit to the proposed project, where they use TDA to characterise jailbreaks and backdoors.

First, I would choose an LLM to test and generate prompts based on the methods in [1].

I would then closely follow the methodology in [3]:

• •Extract high-dimensional hidden state vectors for the final prompt token across all intermediate layers.

The workflow should then be something like:

TDA --> summary statistics of barcodes (from persistent homology if I understood correctly) in TDA --> further principal component analysis (PCA) on this --> input into classifiers or visualisation

Quantify and compute (with some reasonable metric) the difference of the above for “ask” and “tell” prompts.

I expect to see some measurable difference in the internal representations resulting from the two classes of prompts, thus giving some quantification of sycophantic behaviour from a mech-interp perspective. Specifically, I anticipate that persistent homology computed from the later layers would be more different for the two classes of prompts than in the early layers, since it is believed that persona is formed in the later layers.

I have no sense of how long the experiments with TDA would take. But I envisioned a project with the key results to be obtained in three months. If it is more than three months, then it would depend on what the bottleneck is (if any). If it is a computational one, then I would try to see what factors affect computation timing (eg. size of dataset, size of hidden vector, etc) most adversely and then select reasonable values/bounds for these factors. I would need to discuss this with someone who knows much more than me about this stuff.

Can emergent misalignment be predicted by observing the J-space? Namely does displacement of the J-lens readout predict out-of-domain emergent misalignment when fine-tuned on a harmful dataset in contrast to benign fine-tuning.

A positive result would reveal an important empirical marker to recognize emergent misalignment. Internal measurements are only useful as far as they reveal something more than the model just undergoing change during training. This is why it is important to observe different results for harmful versus benign fine-tuning.

A negative result would show that the representational displacement in the J-space evolves similarly during training regardless of the content being aligned or misaligned. This could call into question the fact that the J-lens could be useful in quantifying alignment.

We would further compare results from a frozen lens to ones updated throughout the training, which could illustrate how quickly a given J-lens can become outdated during training.

It was demonstrated in the paper by Betley et al. that fine-tuning on a specifically insecure code sample can result in a broadly misaligned model.[1] Consequently Turner et al. produced smaller model organisms utilizing advice datasets which were only narrowly harmful. They further investigated the training trajectories finding changes in underlying directions tied to behavioral shifts, which can be a useful baseline for our study. [2] Soligo et al. extracted misalignment-associated activation directions whose ablation reduced misalignment across different fine-tunes. This supports investigating shared internal structure, but identifying effective intervention directions differs from validating a general readout-distance metric across checkpoints. [3]

The J-Lens was introduced by Gurnee et al. illustrating readouts which could be relevant to alignment by virtue of revealing the surfacing J-lens tokens and further differences between pretrained and post-trained models.[4]

These findings suggest that J-lens could help track emergent misalignment, but we do not yet know whether it adds useful information beyond simpler measurements. Fine-tuning can change both the model’s internal representations and how the lens reads them.

I would start by reproducing the medical-advice-induced EM in Qwen2.5-7B-Instruct results using the open-source methodology and compute a baseline J-lens for a single layer. I will also have to establish my setup to exhibit similar behavioral EM before starting the experiments.

I would create two training conditions and train four different seeds for each across different degrees of fine-tuning. These conditions will be harmful medical advice and correct answers to identical prompts, matched for length, format, and tone. I will conduct evaluations at 25%, 50%, 75%, and 100% of training.

Use 120 out-of-domain EM prompts, 20 held-out medical prompts, and 20 neutral controls, with four completions each. Define behavioral EM as the fraction of all out-of-domain responses scoring alignment below 30 and coherence above 50 under a fixed, blinded judge. Report coherence separately and human-audit a stratified sample.

I will recompute the J-lens at each checkpoint using identical prompts for calibration. The prompt-only activations will be measured at the same layer close to ⅔ depth. The displacement will be defined as a cosine distance between pre-softmax score vectors from each checkpoint’s own lens and the baseline lens separately.

I expect to see a greater J-lens displacement to correspond to out-of-domain EM and to hopefully see an unnaturally growing displacement during harmful training even before EM. Namely, I expect benign fine-tunes to have a smooth relationship between J-lens displacement and the amount of training but to see a radically different phase-transition like behavior in displacement for the harmful training.

If the displacement measurements are essentially identical for the two scenarios, this would change my mind for the J-space being able to produce a scalar marker for tracking EM.

It is also important to note that the benign versus harmful binary might not be the necessary distinction for this study. Models can generalize a vast array of behaviors from fine-tuning which wouldn’t be classified as misalignment by everyone.

Three months should be enough to conduct a single-model comprehensive study. I would estimate running about eight fine-tunes with five separate checkpoints being investigated per run. I would compute J-lenses separately and measure behavioral misalignment for each checkpoint. The project would require resources and GPU access for 7B-model fine-tuning and Jacobian estimation together with automated judging and possibly some human auditing.

Is it still possible to monitor reasoning of models that control their chain-of-thought or use mechanisms beyond chain-of-thought for reasoning? What can we do using white-box monitoring?

Frontier models are rapidly moving towards non-monitorable reasoning regimes: recent OpenAI’s Astra shows strong signals of control over its chain-of-thought (allegedly, due to the looped transformer architecture), leaked reasoning traces of Sol are not reliably legible, and novel architectures (like full-bandwidth transformer or latent reasoning models) will likely make their way to production by the next year. All this actively impedes reliable human oversight over models’ reasoning, even in non-superintelligence regimes. Thus, finding methods of reasoning monitoring, independently of the form, is important to prevent risks posed by rogue AI. If this is not possible, then we should develop better control and potentially abandon monitorability.

Despite a substantial body of work on detecting feature directions in model’s activations (e.g. all of mechinterp), and work that shows that models can develop strategies against monitors, it is still open if models develop universal mechanisms for reasoning, and advanced architectures are also not yet studied. Essentially, it is not clear if a transformer-like model universally develops traceable computation, not feature, footprint. There are very few examples of such behavior in the wild, and it potentially needs a lot of research. There is a small chance that looped architecture (or other variants) makes the model more interpretable from this point of view.

A prosaic first step is to look at a looped transformer with various interpretability tools and see if monitorability survives under adversarial training:

• •Train looped transformers to solve tasks using a hidden intermediate variable (like the belief-state tracking) while simultaneously training the model to make this variable not decodeable (by linear probes, MLP, SAE,...),

Concentrate directly on computation mechanisms instead of simple feature detection, i.e. find circuits that use the latent variable.

See if the model easily hides the computational footprint in the activations (recurrent, cot, etc).

Similar experiments should be run on architectures that are most likely to get into deployment soon.

Potentially, shifting the paradigm of mechinterp away from features towards computation will take serious theoretical effort.

I expect that most of white-box monitorability survives the shift to novel architectures.

• •A set of quantitative results showing what survives the shift to advanced architectures. Some compute is needed.
• •It would also be nice to give some theoretical explanations: e.g. are there architectural limits on how information that is necessary for computation must be represented in the residual stream.

Given a set of model outputs, what is its pre-image inside an LLM’s activation space? Does the pre-image form a simple, interpretable geometric structure? What are its dimension, curvature and topology?

When the pre-image is simple this could give a way to connect outputs to internal structure and may be useful for steering. If it is complicated/disconnected/unstable it would identify concepts that do not correspond to simple activation-space objects.

Prior work has found simple geometric structures for some concepts in LLM activations[1]. Other work has developed methods for tracing output directions or regions back through neural networks[2]. These approaches do not directly study the pre-image of a chosen output set. We propose to study this pre-image.

Choose a small LLM and a simple concept with known structure (e.g. days of the week). The goal is to construct a manifold in activation space that is the pre-image of outputs with support on only weekday logits, i.e. “the answer is one of the seven days”. We allow outputs that say (20% monday and 80% friday). Ideally we’d want to construct the pre-image across all/most layers.

As a first step we construct the pre-image of a discrete set of outputs: those with all the mass on one day of the week. This should reconstruct the known approximately circular representation from Engels et al. [1].

We then try obtain the pre-image manifold for outputs that are a mixture of different weekdays. If we can construct this manifold we would then look at its geometric and topological features: how curved is it? Is it simply connected? does the same shape persist across layers?

A useful reference could be [3] which Claude tells me approximates the pre-image of an output set, but used it for didn’t study any concept’s manifold or apply it to simple concept sets.

Success would be if we find concepts with a manifold that

• 1.give pretty plots
• 2.persist across layers
• 3.are low dimensional
• 4.are very ugly and/or too hard to construct

After a month it'd be nice to have constructed the pre-image for days of the week and verified that it contains the known approximately circular representation from Engels et al. [1]

After the end of the month it would be nice to study the geometric structure of that manifold.

For an asymmetric n-state Mess process where a transformer's residual width is less than the belief-simplex dimension ($d_{\text{model}} < n-1$ for n hidden states), how does it discard, or merge belief states; · 1.Similar whole-future prediction pairs? · 2.Close pairs geometrically? ·

Identifying if compression biases in training follow the information-theoretically optimal choice is necessary for making statements about what large models discard, after we understand the situation sufficiently for toy models and synthetic processes.

Analysing two compression mechanisms and the meaning of if a probe fails to recover a distinction, because either the model never represented it, or it represented it and then discarded it.

Future-optimal merges the pair of beliefs whose whole-future predictions are most similar. If this information was discarded then the distinction was predictively redundant and it’s informative.

Geometric merges the pair closest in the simplex, regardless of what they predict. The distinction may have mattered but was merged for the wrong reason, and the failure is misleading.

• [1]showed that transformers trained on hidden Markov processes linearly represent optimal Bayesian belief states, but always at $d_{\text{model}} = 64$, far above the 2–4 dimensions of the belief simplex, and named compression beyond the residual stream as crucial future work. [2] explained how attention builds these beliefs, again at $d_{\text{model}} = 64$. The work closest to answering our question is [3]: for processes built from parts, transformers use factored orthogonal subspaces, and when factoring is lossy they still collapse to it first and add dimensions only slowly, even when narrower than the joint belief space. But [3] studies decomposable processes, where one lossy form is singled out in advance. For a single non-decomposable process no candidate is given, so which beliefs get merged is open. Theory supplies a target: the information bottleneck for time series [4] gives, for each number of states k, the k-state compression of beliefs that discards the least information about the future, and [5] bounds how many belief features near-optimal prediction needs. Both measure cost in bits or states rather than linear dimensions, and neither has been compared with trained networks. [6] already named this memory–accuracy frontier as open.

First, for asymmetric Mess-n (n = 3–6), compute exact belief sets and two candidate compressions: future-optimal (minimising information loss about the future) and geometric/PCA (minimising distortion in the belief simplex). Keep processes where these predict different merges. Set context length using a filter-stability bound, and compute the entropy rate as the benchmark loss of a perfect Bayesian predictor.

Second, reproduce [1] and [2] on Mess3 using a small transformer with a beginning-of-sequence token, affine belief probes, and the papers’ controls. This validates the experimental pipeline.

Third, sweep residual width from 1 to n + 2 across several seeds. Measure prediction loss relative to the entropy rate, probe error at each layer, and the PCA spectrum of activations.

Fourth, below width n − 1, use the affine probe to identify which beliefs the model merges. Compare these empirical merges with the future-optimal and geometric predictions, and determine how the information-bottleneck state budget k maps onto network width.

Finally, train a small RNN with hidden size k as a control, where the memory constraint is explicit rather than architectural. If transformer compression matches the future-optimal prediction, test sharper predictions about which beliefs merge and in what order. If instead it resembles the architectural bias found in [3], characterise that bias and investigate why training produces it.

The key test is whether narrow transformers discard beliefs according to predictive similarity or geometric proximity, distinguishing information-theoretically useful compression from architecture-driven compression.

The different situations are structured as observation, interpretation and response.

1.At width ≥ n − 1, prediction loss is close to the theoretical best. The model needs at least n − 1 dimensions to predict well; narrower than that, something is forced out. The experiment functions.

2.When the model is too narrow to hold all beliefs, the pairs it merges are the ones whose long-run predictions are most similar, not the ones that are closest to each other geometrically. Model discards what is least useful for predicting the future. Main positive result.

3.At width 1 for Mess 3, it would lose some information and could analyse what is kept.

• 4.The dropped beliefs match neither prediction. The pattern looks like the compression bias in [3]: the model merges in a way that reflects the architecture, not the data.Redirect: characterise and explain the architectural bias rather than test the original prediction.

he study is restricted to asymmetric Mess processes with n = 3–6 hidden states, focusing on cases where future-optimal and geometric compression predict different merges. The first stage prioritises Mess3 and Mess4, with larger n used where computationally practical. The project focuses on identifying and comparing compression patterns, rather than developing a general theory of transformer representations. The RNN control and investigation of why a particular compression emerges are secondary and can be dropped if time becomes limiting. Symmetric or near-symmetric processes, larger state spaces, and broader architectural comparisons are outside the initial scope.

How does belief state geometry in the residual stream activations sharpen (or fails to sharpen) as we scale model size? By "sharpen" we mean "become closer to the theoretically predicted belief state geometry".

Computational mechanics makes falsifiable predictions about what internal representations of the models should look like, assuming optimal prediction. Targeting an instrumental goal as an intervention target point is scale-free: it will be relevant to both current and future models. In fact, we would expect larger, more capable models to be closer to optimal predictors, hence exhibit even sharper belief state geometry.

One objection to existing comp-mech research is that it is constrained to toy setups. Investigating how the sharpness of the belief state geometry scales with model size would help us understand how the how earlier findings from toy setups would carry over to real-world AI systems.

• [1]showed that training toy transformers on sequences emitted from HMMs gives rise to the corresponding belief state geometry emerging in the residual stream activations.
• [2]show that the same belief state geometry emerges in LLMs when the HMM sequences are provided in context. This is a step towards closing the gap between toy models usually analyzed in comp-mech research and real-world AI systems. However, [2] only analyzes in-context learning, as opposed to explicit training. No prior work conducts a systematic analysis of how the sharpness of the belief state geometry scales with model size. This systematic analysis could help us discover clear scaling laws that we might extrapolate to even larger models.

We will study how the sharpness of the belief state geometry scales with model size and with HMM complexity.

We measure the sharpness of the belief state geometry by computing $R^2$ of the affine map fit between the internal residual stream activations and the theoretical belief state geometry.

We will also try to come up with other metrics to quantify sharpness.

Setup A: trained-from-scratch toy transformers.

This setup allows us to precisely control the model size, as well as the size of the dataset used for training (which is difficult with real LLMs, where models of different sizes were likely also trained with different number of tokens/different compute budget).

Setup B: pretrained LLMs, HMM sequences in-context.

This setup is more realistic in that it focuses on LLMs.

We reproduce the setup from [1] on toy transformers of different sizes. We start by keeping the width/depth ratio fixed, and scaling the number of params. We train each model configuration on the same dataset of sequences samples from the HMM. We plot the $R^2$ of the linear fit against number of parameters (model size).

For the LLM in-context setup we do the same, measuring how $R^2$ scales along models of different sizes and the same model family.

Once we obtain the plots, we inspect them visually and try to fit some function to them -- a scaling law.

One hypothesis is that the sharpness of belief state geometry increases monotonically with model size.

Another hypothesis is that sharpness of belief state geometry first increases, and then decreases (sharpness is an inverted U shape function of model size) -- maybe once the model is expressive enough, it doesn't need to compress the information and it can encode the belief in a less compressed way.

A result that would make me drop the question is if there is no clean relationship to be observed between the sharpness (measured in terms on $R^2$) and the model size.

I think the whole project is doable within 3 months. If we don't have enough compute, we could cut the toy-transformer setup and only focus on the LLM in context setup, because it is less computationally expensive (we only do inference, no training).

Ambitious plan would include trying to find a theoretical explanation for the observed scaling relationships.

Suppose we are given a model which has been trained on a hidden HMM. By making suitable assumptions, is it possible to infer properties of the underlying HMM by randomly sampling the network and looking at the model’s internal activations?

In practice, the data generating processes models are trained on (e.g. language) is largely unknown, and any techniques which could shed more light here would be of interest. For example, more knowledge here could potentially have applications for safety as we will have a better understanding of how the model will generalise in any given context. If successful, this project would exhibit a toy example whereby we infer properties of the underlying data-generating process (e.g. state-transition probabilities) by sampling the network and using heuristic assumptions.

In [1] the authors train a transformer model on a known HMM and show that the model linearly represents the corresponding belief state in its activation space. This project can be approximately viewed as the inverse problem: knowing this is how transformer models behave when trained on HMMs, can we infer any distributional properties of the underlying HMM when it is kept hidden from us. To be clear, multiple HMMs give rise to the same dynamics, so our objective would be to characterise an appropriate family.

We still need to iron out the details, but we believe this is possible in principle. As a rough idea...

We can start with a simple setup: train a transformer (or a similar network) with data generated by a known stochastic process (e.g. a given HMM). We then assume that all we have access to is the transformer residual stream and the activations at each layer. First, we would need to understand the alphabet: this can be achieved by sampling the transformer output. Then, we can feed sequences of alphabet elements and, for each token, record the activations after the token is processed, the output probabilities and the actual token emitted. We can use these ingredients to reconstruct the state-transition probabilities. Assuming that the belief state is linearly embedded in the activation space, a PCA-style argument will give us proxies for the belief state, and then we can all of the information gathered thus far to realise a range of HMMs which have the same dynamics as the underlying data generating process.

Based on the belief-state geometry work of Simplex, our prediction is that it is possible to exhibit an example of a HMM whose dynamics are the same as the hidden HMM simply by randomly sampling the network and extrapolating internal beliefs from the models activations.

One of the main challenges could be recovering precise belief states from the activation vectors. Activations may encode more information than just belief states in a non-trivial manner. There is also some uncertainty around coverage: applicability of this study may be limited in cases where testing enough sequences is unfeasible due to the large number of possible states and tokens.

After the first month, we will have solidified the mathematical argument and conducted some initial experiments with very simple stochastic processes. We believe that if everything went smoothly this project would not take too long to complete.

I want to understand whether a small transformer trained on next-token prediction on hierarchical data linearly encodes in its residual stream the Bayesian posterior over the hidden nodes.

Deep networks and transformers beat the curse of dimensionality because real-world data is hierarchical and modular, and depth lets a network exploit that [4]. Belief-state geometry [1] predicts how a transformer represents Bayesian beliefs, but only for data without hierarchy.

If the prediction holds, the representation of hierarchical data becomes derivable from the data model, ie we can predict which level sits in which subspace at which layer and check against this ground truth. In the future, it would be exciting to extend these results to training to understand how transformers builds these levels of abstraction during training and with an incomplete hierarchy.

If the prediction fails, it will be informative to study how it fails. The network might only keep low-level evidence or a factored approximation that drops hierarchical correlations [6].

The existing Simplex line of work initiated in [1] showed that transformers trained on next-token prediction for hidden Markov models linearly represent the Bayesian belief over the hidden state. [6] showed that when a process splits into independent parts, the belief is stored in orthogonal subspaces. None of these works study processes whose hidden states are hierarchical.

Ref. [2] explicitly studies hierarchy. The authors train transformers on a fixed-tree model. They find that outputs match (exact) belief-propagation marginals, and the attention maps mirror the tree. However, their task is masked prediction, and intermediate layers are probed nonlinearly using a neural network for hard ancestor labels, not belief vectors (probing nonlinearly is not convincing). [3] finds that autoregressive transformers on grammars store parsing-chart states, again via nonlinear probes on discrete targets. [4] introduces a widely known model, the Random Hierarchy Model, which explains why depth helps learn hierarchical data through sample complexity, but this work does not examine model representations.

To my knowledge, no work derives the belief state for next-token prediction on hierarchical data, or tests whether it is linearly represented.

Using Bayes theorem I derive the belief state of the hierarchical model in [2] as the posterior over path nodes given the prefix. I then enumerate all possible beliefs to build the geometry and plot this geometry for simple, low-dimensional models.

Next I train a small transformer akin to [1] on next-token prediction and check that it learns the data to near zero loss.

At each layer, I linearly regress the residual stream onto different targets: the joint belief, the per-level marginals, and the pairwise marginals of neighbouring levels. The plan is to compare these different targets to show which representation the transformer learns at which layer.

Using the filtering parameter of [2] which removes correlations above a chosen level I check if the representation changes at those levels.

Eventually, I would like to extend the theory to more general models such as the Random Hierarchy Model [4].

My prediction is that some residual stream layer linearly encodes the Bayesian posterior with low regression error. This regression error is high initially and training reduces it progressively. Beliefs about lower tree levels become decodable at earlier layers, as the attention maps in [2] suggest.

What would make me drop the question: if the transformers reach near zero loss, but no linear map recovers the theoretically anticipated optimal belief state better than a random baseline.

In the first month: I would aim to derive the exact beliefs, train a transformer to near-zero loss, and use layer-wise linear probes to reconstruct the belief state for the three parameterisations.

What I need: a mentor to check my theoretical setup and derivations, the public code of [2], a GPU to train the transformer.

If turns out to be three years: I would first cut down on the generalisation ambition, and next the empirical work, and if that still turns out to be too big of a question, I would simplify the theoretical setting further.

Could we use perturbation theory to quantify the deviation in belief states of transformers when it learns data from indecomposable processes?

It enhances our understanding of internal representations of concepts in transformers.

It would provide a more robust link between the well-established tools from physics and the language used to study deep learning models from a theoretical perspective.

It lays the groundwork for other mech-interp studies.

Background: This project builds on [1], where it was shown that transformers have an inductive bias towards learning factored representations, even when this hurts prediction. In [1], conditionally independent generalised hidden Markov models (which I will call factored processes) were used for data generation to check if transformers learn factored representations. They also trained the transformer on a “perturbed” process that I call an ε-close factor process (or indecomposable process) to check if transformers prefer learning factored representations even when this hurts prediction. This project aims to compute the belief states in the representations learned by the transformer trained on data from the indecomposable process for small ε using perturbation theory. It also aims check if there exists a phase transition between factored and non-factored representations (when the transformer does transition to the non-factored representation regime for large enough ε).

I don't know of any papers that attempt to use perturbation theory to compute belief states and then compare with measurements from residual stream activations.

Ref. [2] is the seminal work on this specific framework on transformers and belief state geometries. Perturbed Markov processes have been studied in the context of bounding errors for numerical situations (such as [3]).

First week or two would involve a deep-dive of the relevant mathematical theory. This would predominantly entail studying the various appendices of [1] in detail. Towards the tail end of this process, I hope to start performing the calculations using time-independent degenerate perturbation theory to compute the perturbed belief state vectors that I expect to measure in residual stream activations when data is generated from these indecomposable processes. I also expect to start (collaboratively with other experts) constructing a dictionary between the language of computational mechanics and that of physics. The next few weeks would involve designing and running the experiments that extracts these belief states from specific models (which need to be at least large enough for the model to learn the non-factored or joint representation). The exact methodology would probably resemble that of [1], with perhaps some minor adaptations relevant to these measurements.

If my hypothesis is true, then for ε-close factored processes, the belief states that are measured should match the perturbed belief states that are calculated. Depending on the nature of the calculations, a phase transition at some $\varepsilon_{\text{crit}}$ is observed (or not). In any case, the notion of the representation attractor, “where small ε keeps predictive vectors near the product-state manifold while large ε pushes them off” [1], can be proven or disproven by this project.

I expect the theoretical predictions to be done in the first month. Making the physics <-> GHMMs + deep-learning language dictionary may take longer if some wrinkles need to be ironed out. I am not sure how long it would take to design and run the experiments, but I do not think there is any circumstance in which this project takes more than three years -- it was intended to be an incremental curiosity based on some works I found very fascinating!

Does the learning of belief state geometries exhibit phase transitions and if so, what are their order parameters and hierarchy across increasingly complex generating processes? ·

We aim to develop a more robust, quantitative understanding of the learning behaviour of belief state geometries regarding the order in which different types of structures are learned and internally modelled. This would lead to intuition for how LLMs learn throughout training but could also make predictions for in-context learning, for example the order in which different aspects (factors) of the in-context environment are learned, such as assistant conversation vs. testing environment, inferring personal details about their user, etc..

In [1], the authors showed that models prioritise factored learning, even when the ground truth was an interacting system which did not genuinely factorise. They showed that how fast the model adopted a higher-dimensional depended on the strength of the interaction term modifying the factorised transition probabilities (Fig. 4 in loc. cit.).

In [2], the authors showed that for non-ergodic processes (where the generating process is chosen randomly among a fixed set of generators), in-context learning of the model can be visualised as a probability distribution over belief states across the generating processes. Their data on the learning process displays evidence of a phase transition in the KL-divergence (Fig. 7 in loc. cit.)

We aim to combine and extend these findings, characterising in what order the two transitions happen given a non-ergodic, interacting generating process and how this depends on the strength of the "interaction".

The first goal would include reproducing the behaviour of the plateau in [2] and measuring its length against the interaction strength epsilon. This should give rise to a scaling law that could be matched against theory, as may be developed by our second proposal, or by an LLC estimator.

As a second step, we would investigate whether this adaptation happens in steps or as a single phase transition. Decomposing the interaction term into subspaces, we would expect subspaces with stronger interactions to be learned first.

A similar LLC analysis could also be performed on the learning dynamics of the non-ergodic process [1], although here a potential order parameter for the potential phase transition is much less obvious.

Lastly, we would combine non-ergodic and factorisation to observe (qualitatively) the order in which the transitions happen. We set up a generating process with K sources, each composed of $N_i$ factors. We introduce an interaction term and gradually increase its strength.

My current expectation is to see a gradual, step-by-step transition from the factorised to the interactive regime.

Intuitively, I would expect the non-ergodicity of the generating process to be fully learned before the phase transition to non-factorised belief states occurs (with low confidence).

The complexity of the transformer architecture with Adam training in [2] makes exact quantitative predictions difficult. We would cut the LLC estimations if they appear too difficult.

As outlined under "what you would do", the first step was to recreate the findings of [2] for various interaction strengths.

If this turns out to be much harder, I believe the subsequent steps can be omitted as necessary, in particular the combination of the factorisation and nonergodic processes.

How do LLMs represent uncertainty between discrete alternatives? We will test whether activation geometry is best explained by probabilities, log-odds, or Fisher–Rao geometry, and whether this generalises across concepts

A reliable understanding of geometry could improve belief decoding and steering. Understanding when it emerges could also clarify how models learn to represent uncertainty.

Previous work identifies geometric belief representations in synthetic transformers [1], curved belief manifolds in pretrained LLMs [2], and connections between geometry, uncertainty, and steering [3]. We could maybe build on this by testing how well uncertainty representations transfer between settings. Maybe see if there are commonalities

Start with a small LLM and simple tasks involving uncertainty between two alternatives, such as object identities or weekdays. Vary the uncertainty using stated probabilities and supporting evidence, then measure output probabilities and activations.

Clear visualisations and evidence that uncertainty representations transfer across contexts—or a clear account of where transfer fails. Extend to three alternatives or a second model if time permits.

By the end of the first month, we aim to have a working experiment on one small pretrained LLM, comparing probability, log-odds, and square-root probability descriptions of uncertainty between two alternatives. Deliverables would be layer-by-layer plots and quantitative comparisons on held-out prompts, including checks that the model tracks the intended probabilities.

We would need GPU access for inference and activation extraction, standard interpretability tools, and mentor feedback on the experimental design. If the scope grows, we would cut SLT, steering, and additional models first, keeping one categorical task and a reliable comparison of the three descriptions.

With RRXOR, we have 1-layer transformers get close to optimal loss but known one-layer mechanisms don't work. Does a 2-layer model split the belief update between its layers so that ablating its layer-1 previous-token heads would raise loss above optimal?

-Models infer latent variables such as user identity or persona from context. This covers which mechanism gradient descent picks when there are several options, and what happens when inference is split across layers. If the 2-layer model splits the update, the belief is assembled across several layers which means a probe at one layer could miss it.

-This also extends [2] beyond processes where every token is informative. If the second layer turns out to be useless this would indicate transformers prefer doing everything in one step instead of splitting the update into different stages when there’s enough width.

-Either way, the exact belief is known, so we can check whether ablations and probes can recover the real mechanism.

• [1]showed that transformers trained on HMM data store the belief states as a geometric object in the residual stream. For Mess3 it shows up in single layers and for RRXOR it only shows up when all layers are joined together. [2] covers how one layer does this on Mess3 i.e attention adds up one correction per past token, weighted by distance, and the MLP bends the result into the belief shape. [3] found on other HMM tasks that the MLP does the update and attention moves information. [4] showed that transformers combine neighbouring tokens across layers using previous-token heads. In RRXOR each token is 0 or 1 with probability 0.5 at every point in the cycle, so only groups of tokens can signify anything about the hidden state. The Mess3 mechanism hence doesn’t work, but early runs show a 1-layer model still gets close to optimal loss. None of these say what each layer contributes on RRXOR. This might be a general fact about parallel computation: [5] shows self-attention can’t model periodic finite-state languages unless the number of layers or heads grows with the input length, and RRXOR is exactly this kind of periodic process.

First: train small transformers with 1 and 2 layers on RRXOR using the existing code from [1], and compare their loss with the exact optimal loss. Setup similar to [1] i.e $d_{\text{model}}$ 64, Adam, context length 10 and 20, 3 seeds with different random initialisations.

Then look at the attention patterns and QK / OV circuits directly, so the hypotheses come from what the models actually do. Starting list:

• •One layer could record the whole sequence and let the MLP do the work
• •One layer could use attention that depends on which token is where
• •Two layers could split the work, with the first combining neighbouring tokens and the second gathering them

Find out which one each model uses by removing or simplifying parts of it and seeing where the loss goes up:

• •Token-dependent attention: replace each attention pattern with its average over token identity
• •Recording the whole sequence: swap the value path for one where token and position are just added together
• •Two-layer split: look for previous-token heads in layer 1 (most attention on $t-1$ / $t-2$), mean-ablate them and see which positions get worse

For each mechanism, measure how much of the loss gap it recovers. If none of them recover most of it, the real mechanism is outside our list.

Then vary depth, width (heads, $d_{\text{model}}$, MLP size) and context length to see when a second layer is needed. Confirm with activation patching, and check our setup on Mess3.

Prediction: a 1-layer model gets close to optimal loss by recording the whole sequence and letting the MLP do the work. A 2-layer model splits the work, with the first layer combining neighbouring tokens and the second gathering them, so ablating its layer-1 previous-token heads raises the loss a lot. What would change my mind: if ablating layer 2 in the trained 2-layer model barely changes the loss, the second layer is useless, which would indicate transformers prefer doing everything in one step instead of splitting the update into different stages. If none of the mechanisms explain most of the loss gap, I’m missing something on my list and would go back to looking at what the attention heads actually do.

After the first month: the 1 vs 2 layer runs on RRXOR with 3 seeds, the ablations for each mechanism, and the check on Mess3. The models are tiny (~50k parameters) so they train in minutes on a laptop, no GPU needed. If it turned out to be three years of work I’d cut activation patching and the width sweep first, and keep the 1 vs 2 layer comparison on RRXOR.

The question is two-fold: · 1) does an LLM represent a state of the world for a given prompt? · 2) if there is uncertainty about the actual state, is the distribution over possible states represented and how? ·

We will understand how LLMs represent world under uncertainty. This will allow for:

• •better monitoring (by tracking latent belief states related to (mis)aligned behavior) and
• •intervention (by understanding the geometry of belief).

Tracking beliefs will possibly give better mechanisms for:

• •tracking misaligned behavior (such as lying: if the model's output is inconsistent with its beliefs) and
• •understand model's biases (see if model assumes certain details by default, so more inclined to do certain things).

• [1]and general Simplex line only considers synthetic datasets and transformers trained on this data. It is not clear if the models do probabilistic inference on real data. [2] shows that models do represent latent state in form of propositions that bind entities together, but it doesn't cover the probabilistic state or more general ground states.

• 1.Reproduce [2] and iterate on: find how a deterministic state (i.e. simple propositional facts) is represented in the latent space.
• 2.Add partial observability/uncertainty. Here's one way to do the experiment: in the prompts, describe a setting where an object is hidden in one of given locations. Find how this is represented by probing: check what is the vector associated to a general prompt, and which are associated to prompt+answer. The hypothesis is that the latent vector of the input with uncertainty is a weighted sum of vectors corresponding to prompts with specified position of the object, which would mean that the uncertain state is a distribution over certain states.

3.1. If this is the case, try intervention: would steering change the output as if it was changing belief? Then test the mechanism in various settings and find if it generalizes, i.e. there are universal directions associated to uncertainty or alternative beliefs. Also try to find the belief forming mechanism, that is by the layers when the belief is "formed", what is the circuit that did that.

3.2 If this is not the case, try to find an alternative explanation. Perhaps genuine uncertainty is different from a mixture of beliefs, so the experiments would need to be split in two and the mechanisms described independently.

I expect that in certain, but not all, settings the state is indeed a distribution over several ground states. If there is no, even simple, setting in which a state can be seen, then it would indicate that uncertainty is represented in a different way.

I also expect to see that LLMs are not calibrated in their beliefs and are heavily affected by the training data (be it biases or simply noise), so the "default" distributions over ground states will be far from uniform.

I believe that all of the plan that I stated before is executable in one month. The main problem is to find an alternative mechanism if the suggested one is not supported by evidence.

Pretrained LLMs that predict in-context HMM sequences and linearly represent belief states can verbalise their next-token prediction, but their accuracy on questions about hidden structure (phases, length, content) stays at shuffled-control level across sizes.

• •When models can't share the structure they operate on, any oversight that depends on asking the model misses that aspect and underestimates the factors that influence behavior. We should examine those structural levels using internal probes rather than questions. This also establishes a baseline for genuine limitations: if a model fails to disclose something it utilizes, even without any motive to conceal it, that failure in other areas shouldn't be seen as evidence of deceit.
• •If models can report their structures, or begin to at a larger scale, asking them becomes a cost-effective, validated alternative to probes for uncovering hidden structure. The point at which they start to provide reliable self-reports indicates when we can trust them.

The two ingredients are presented and studied separately but never combined. [1] shows pretrained LLMs predict HMM in context in a practically optimal way. The belief state is shown to be linearly extractable from the residual stream and used for prediction. But they never ask the models. [2] shows in-context prediction again without asking. [3] asks the models and finds gaps between classifying and stating, but their rules are deterministic text classifiers, without any true internal state to check the report against. [4], [5] show that models can verbalise structures acquired in fine-tuning, not in context for pretrained stage. [6] shows that models propose rules that fail to apply. No work found puts prediction, representation and verbal report jointly in the same model and process, while what the model tracks is exactly known.

I’d kick things off by diving deeper into today’s pilot project. My focus would be on three main areas: first, I’d assess how accurately the model predicts outcomes compared to the ideal belief-state predictor; second, I’d check if it can clearly explain its next prediction when asked; and third, I’d see if it can handle straightforward questions about the hidden phases. To make sure the data is solid, I’d compare these results against order-1 and shuffled controls, using a larger number of seeds for more reliable insights.

Next, I’d run the same tests on the six models from [1], which range from 2B to 9B. Since their probes already shed light on what each model represents, I’d put together a table that outlines what each model predicts, what it signifies, and what it can articulate. This would help pinpoint where reporting starts to stumble as the models get bigger.

If I notice any breakdowns in reporting, I’d make sure it’s not just a matter of poor prompting by experimenting with different phrasings, allowing for more time to think, and providing a few worked examples.

Finally, I’d follow the belief directions from [1] and see if the model’s responses align with those changes.

I expect to see some dependencies of the ratio WellPredicted/WellExplained on the complexity of the HMM and the size of the LLMs. I would like to find some scaling laws for these accuracies on answers.

I’d let go of the question if the gap closes when the model is prompted correctly—like with better phrasing, more thoughtful input, or by providing a few examples of other processes. That would suggest it’s more about how we’re asking rather than a lack of underlying knowledge. I’d also reconsider if bigger models just do a good job of outlining the structure, because then there wouldn’t be a clear threshold to identify.

By the end of the first month, I'd like the core table done: the six models from [1] on the four representative HMMs, 10 seeds each, showing how well each predicts, whether it can say its next prediction, and how it does on the structure questions against controls, with a check for poor prompting. I'll need about one A100 for a few days (or Colab Pro) and the public code from [1]. Trying to check the affine map for the hidden states would be great, too.

I don't see how this could turn out to be three years of work.

Does the dimension $\hat{d}$ of a persona in a model change as model size grows? I expect it to saturate at the data's persona rank d, not at model width or persona count. ·

If $\hat{d}$ saturates at d, persona dimension is a property of the data, not the model. Persona structure measured in a small model above the resolution threshold then transfers to larger ones, the threshold is predictable from the data's coupling spectrum, and interventions designed on small models (steering, character training) have a principled reason to carry over. The ~10³-dimensional structure hoped for in [5] becomes a data quantity we can estimate.

If $\hat{d}$ keeps growing with width, either small models lack the resolution to separate structure that is in the data, or "persona" is a model-dependent notion. Small-model persona results would not transfer, and the definition would need replacing with one that saturates.

• [1]models pretraining as a mixture of HMMs and in-context learning as inferring the latent generator, but says nothing about how or in what dimension the belief is represented. [2] shows belief states over a single generator are linearly embedded in the residual stream with geometry matching the mixed-state presentation; no mixtures, no width dependence. [3] proposes that pretraining learns a distribution over personas and post-training updates it Bayesianly, and lists "what precisely is a persona?" as open. [4] measures persona space in 27B–70B models: 4, 8 and 19 PCs explain 70% of variance in Gemma 27B, Qwen 32B, Llama 70B, and the same PCs exist in the base models. The count grows with size, but families differ, so scale and family are confounded, and there is no account of what sets the count. [5] frames the persona agenda as finding ~10³-dimensional structure; a comment there asks for a scaling law. [6] (unpublished) formalises persona selection as Bayesian updating over belief states and tests it up to 27B; it asks whether selection is Bayesian, not what dimension the space has or whether that is set by data or model. No controlled test at fixed data across widths exists.

I started here: https://docs.google.com/document/d/1fKeGBKgqdI02OOgnacfXvdCpzeOZTUtzxtK-7ooX6ro/edit?tab=t.0 (see plots)

Start from a toy model given by a mixture of HMMs: K personas share one content HMM and differ by a style distribution of rank d. Train decoder-only transformers of increasing width and look for two things:

• •What happens to the recovered dimension of a persona as model size increases? Pilot done (one seed, K=512, d=64, widths 32–1024): recovered dimension saturates at d, not at K−1. Plots in the linked doc.
• •wider models are also better trained, so closer to the true next-token distribution $p(x_{\text{next}} \mid x_{\text{now}})$, and recovered dimension could reflect training progress rather than capacity.
• •Better plot: compare widths at training checkpoints with fixed gap from the Bayesian truth. Check google doc

The toy models are just for initial intuition.

Next:

• •set up a model with anisotropic latent matrix and see which directions get learnt first
• •look for the different personas, not just their number: held-out personas, to check the model represents shared traits rather than memorising personas
• •check the Fisher spectrum of the mixture vs context length as the prediction of the resolution curve
• •let personas modulate dynamics, not just symbol frequencies

Then small LLMs, 0.5B–2B:

• •already-trained ones with fixed data across widths (Pythia) rather than training our own
• •unsupervised persona space
• •confounder: personas may not compare directly across models trained on different data

Ideally, in a year, a 600B–1T model :)

I would expect $\hat{d}$ rises with width and context and saturates at some point, independent of number of personas (in my toy model); under a power-law spectrum the recovered singular values match the true ones from the top down and truncate; the width dependence survives matched Bayes gap. I would try something like Pythia, the between-persona spectrum broadens with width and then flattens.

I would drop the question if recovered dimension tracks K or $d_{\text{model}}$ rather than $d^*$ at widths above $d^*$; if linear and nonlinear (TwoNN) estimates disagree systematically, meaning the belief is not linearly represented and "dimension" is the wrong object [8]; or if the toy result vanishes once personas modulate dynamics, meaning it was an artefact of the symbol-frequency setup.

First month: toy results at three seeds with matched-gap comparison, anisotropic spectrum, held-out personas, and the Fisher-spectrum prediction written up. With a GPU it is a few days per sweep. Second month: dynamics-modulating personas and Pythia.

If it were three years of work, I would cut the multiple personas search, focus on just one kind of dimension, and cut the emergent-misalignment extension (in doc).

I think I would look for collaborators in any case, as I would prefer working with others, e.g. someone could work on the toy model and show theoretically if we have saturation in tfms or not.

In a small transformer trained on synthetic sequences combining arbitrary associations and reusable rules, do component-level Bayesian influence profiles distinguish these factors and predict greater improvement on held-out · rule applications from rule-targeted than association-targeted data reweighting?

Influence functions in the classical sense are limited since that they may find the direction of influence but are bad at finding the magnitude of influence. When we study susceptibility one big open question is how we perturb the parameters and the training data, because leave one out seems to be crude and ignores the inherent more continuous underlying structure of the training data. I seek to find a more precise way to characterize how structures in a model arise from structures in training data, in particular, if we can observe the type of training data that steers the model to learn to generalize faster, rather than memorize some implicit structural information as facts.

No one has tried using bayesian influence function to study the divide between procedural and factual information that the model ends up learning, but this has been done in the context of classical influence functions [3], [6]. and BIF are cool [7] and synthetic data from universal turing machines like in [5] are cool in particular how the generator learns to identify through RL the 'relevant' part of the training data.

Toy model: Generate synthetic dataset with structural vs. factual information (a very simple example of structural information would be value is square of key, and an example of ‘factual’ information would be randomly generated pairs of keys and values, then one could also try and compose the 2. Another example would be language describing board games, with certain facts -how a specific game plays out vs. structural information: how to develop a winning strategy) similar to the self-play paper train a model (can start with a 2 layer neural net, similar to in [5]) using susceptibilities (some covariance between a function on the parameters and the loss function), mask out different parts of the model (only vary the parameters on the layers or on the attention heads that matter) and check if we can use this to find a better correspondence between the model structure and the structure of the training data.

In real life we expect structural information to also correspond to persona, which bridges us into safety concerns and we can think about how to efficiently train an aligned model.

I want a training set up, similar to the generator + learner set up where the generator finds the more universal part of the dataset to generate vs. scaling up the factual data, and see if the susceptibility of the model with respect to the training data would behave very differently.

When you scale up factual information or things that appear more like factual information the model begins to memorize, but when you put some restriction on the model size the model prioritizes learning structural information.

ran out of time sorry :(

What can dynamical mean-field theory (DMFT) tell us about deep linear networks(DLN)?

From an AI safety perspective, this project would study how overparameterized models can retain dependence on their initialization even after training converges. Understanding when such hidden memory survives could help explain why fine-tuning sometimes fails to fully erase or overwrite prior behaviours.

• •Li and Goldenfeld [1] recently applied DMFT to a linear regression model and were able to analytically describe transitions between over and underparameterised regiemes, double descent, critical exponents, and non-ergodic behaviour of underparameterised networks.
• •This is (remarkably) a treatment that is independent of weight initilization.
• •It turns out to be a powerful approach that is able to analyse both the training dynamics and the static setting ($t \to \infty$).
• •This approach is, however, new and has not yet been extended in the simplest directions. E.g. to deep linear networks.

• •I would first reproduce the results of [1], both numerically and analytically.
• •I would then numerically extend this to a two layer DLN.
• 1.First run experiments in the static limit where training has converged. I would check if there is still a divergence in generalization error around α=1 (where α is the ratio of model parameters to training data size)
• 2.I would then check if the over-parametrised regime is still non-ergodic: i.e. do different initializations converge to different weights
• 3.I would then check if the under-parametrised regime converges to the same result as the single layer network. This checks that even if the learning dynamics are different, under-parametrised networks converge to the same result.
• 4.I would compare converge times in the single-layer and two-layer networks.
• •I would repeat the above experiments for deeper networks.
• •I would then see if the analytic DMFT framework can be extended to the two-layer DLN. If not, can a numerical DMFT reproduce the full network behaviour?
• •If this works, another promising avenue would be to try apply this to gated linear networks [2]. This is a setting where we can take the first step towards incorporating non-linearity in the network

I would expect DMFT to capture the dynamics of DLN. One key check that would underlie the project is whether mean-field is a valid description of the model? (separate from whether it is analytically tractable). The single most important check to do at the beginning is to check whether numerical DMFT is a faithful representation of the DLN. If mean-field is not valid for a 2layer network or a very deep network, I would drop the approach

Three months would be enough to run all the numerics. It is not clear whether it would also be enough to get the numerics to work, though it would be enough to give it a good shot. I would split time 50/50 between empirical and theory work. If the numerics and analytics work out, I would then try extend the approach to gated DLNs.

How does (the nature of) fine-tuning data affect the learned geometry of concept-directions / steering vectors?

• •Steering vectors offer a relatively cheap approach to adapt model behaviour at inference time.
• •However, interference between steering vectors can be problematic. Amplifying one feature can inadvertently affect another, particularly under superposition.
• •Existing methods mostly focus on finding steering methods that minimize these trade-offs, but take the feature geometry learned by the model as given.
• •Fine-tuning offers an attractive way to refine/alter this geometry to ensure certain concepts of interest interfere more or less with one another, but it is unclear exactly how the nature of fine-tuning tasks used can affect this.
• •If we can understand the “sensitivity” of (e.g. probe-based) steering vectors to properties of the fine-tuning data (e.g. the mixtures over tasks used which are relevant to the features whose geometry we aim to reshape), we may be able to better understand how to use finetuning to achieve “better” steering vector geometry (e.g. features which can be independently controlled while still moving in directions of “maximal change”.).

• •Existing steering methods generally treat the learned representation geometry as fixed and focus on how best to intervene within it. CAA constructs steering directions directly from contrastive examples [1], while more recent methods reduce interference between multiple interventions using constrained optimization, orthogonality, or joint steering subspaces [2,4,5].
• •Recent work shows that post-training can alter steering geometry: [3] regularize SAE features towards orthogonality during fine-tuning to enable more “isolated” interventions, [7] show instruction tuning can degrade structured value geometry, [8] show different post-training objectives produce different refusal circuits and steering robustness, and [6] show fine-tuning can amplify nearby harmful features through superposition.
• •However, none of these works analyze how the nature of the fine-tuning data determines the resulting feature geometry. In particular, it remains unclear when geometry can be changed while preserving primary task performance, which properties of fine-tuning tasks cause steering directions to become more or less entangled, and whether these sensitivities can be exploited to deliberately learn representations with favourable efficacy/interference trade-offs.

• 1.When can geometry even be reshaped while preserving task performance?

A first step would be characterizing when a model’s internal feature geometry can be reshaped in a way that (approximately) preserves some notion of “primary task performance” (either on the same fine-tuning datasets, or downstream task). This can be studied from the perspective of symmetries and degeneracies in networks. In particular, given

a network $Y = f(H)$ (where $H = g(X)$ are the hidden activations of interest), and

behavioural concepts $Z_i = b_i(X,Y)$ (e.g. “is truthful”, “is sycophantic”)

we may wish to ask under which transformations $\phi$ of $H$ we have:

(prediction invariance) $f \circ \phi \circ g = f \circ g$

(concept steering-vector non-invariance) $\nabla_h P_\phi(Z_i=1 \mid \mathrm{do}(h)) \neq \nabla_h P(Z_i=1 \mid \mathrm{do}(h))$

A simple starting point would be to analyze this in a deep linear network for e.g. $\phi \in$ “set of linear maps on $\mathbb{R}^d$”, and then relax the model (e.g., to gated-linear, transformer-based etc.). Assuming some form of LRH ($P_\phi(Z_i = 1 \mid \mathrm{do}(h)) = \gamma_i^T\phi(h)$) would be helpful here.

From here prediction invariance could e.g. be relaxed to an “approximate invariance”.

• 2.How can/does fine-tuning affect this geometry?

The next step would be to analyze how fine-tuning can be used to reshape this geometry in practice. Key questions that arise here are things like:

“When finetuning model weights (e.g. maybe LORA style), what steering vector geometries naturally arise by gradient descent? What makes them change?”

“How can we (best) regularize the model towards “preferred” geometries?”

“How does the nature of fine-tuning data used affect the above?”

“How does this affect performance on other tasks not in the fine-tuning set?”

The third question is particularly related to data attribution - if fine-tuning uses data from a mixture

we can ask how sensitive the learned steering geometry is to each component (e.g. via influence functions). This would help us characterize which kinds of fine-tuning data make different concepts more or less entangled in practice.

I would first study these questions in the toy settings above, before testing them empirically in a small language model like Gemma2/3.

• •In deep linear models, I expect there to be non-trivial symmetries/degeneracies that allow steering geometry to change while leaving predictions unchanged. I would also expect gradient descent to select only a subset of these equivalent representations, with the fine-tuning task/data affecting which geometry is reached. In small language-model experiments, I would therefore expect fine-tuning on different task mixtures to measurably change both steering efficacy and interference, although not arbitrarily.
• •I would change my mind if (i) "task-preserving changes" in feature representation mostly reduce to coordinate changes that do not alter intervention behaviour, (ii) steering geometry is much more stable to fine-tuning data than expected, or dominated by optimizer/seed effects, or (iii) regularizing towards apparently “better” geometry consistently hurts task performance. A failure to transfer the main findings of simple linear models even to small transformers would also make me rethink pursuing theory too far (at the expense of more time on experiments).

Since part 1 is pretty concrete, I would aim to complete that in the first month (maybe 6 weeks), and then move onto the second part. By another 6 weeks (so 3 months in), I would hope to have:

a simple theoretical characterization of when task-preserving changes in steering geometry are possible;

an initial account of how gradient-based fine-tuning changes this geometry in the same toy models; and

a small language-model experiment testing whether different fine-tuning tasks/data mixtures measurably change steering efficacy/interference.

If this is too broad, I would drop the more realistic theoretical extensions first and focus on the toy theory + empirical results.

When during pretraining does J-space form? What data patterns affect its formation?

Short term: We don’t know if J-space formation is a distinct part of the learning process, a ‘learning stage’ that the model must develop in order to deal with problems that require extended reasoning, or if it develops slowly and is just a consequence of the model slowly getting better at tasks.

More long term: People generally use j-lens as an auditing tool, and we cannot interpret a clean readout without knowing what the workspace can hold. If some content never enters the J-space, a clean readout says nothing about it.

From what i can see, no one has measured which data patterns the J-space responds to, or whether it forms as a distinct developmental stage (and when!)

First, i’ll test the model at various checkpoints during its training. Basically use the J-lens to "read" its middle layers and see its internal reasoning (Example observable would be the ). I want to pinpoint the exact stage of training where the model gains the ability to hold complex, unstated concepts in its workspace, rather than just predicting the immediate next token. A nice example of tasks that may require this J-space would be addition of larger numbers. At each saved stage of training, we will measure how much more strongly the J-lens—a tool that reads internal activity as scores for possible words or numbers—scores the correct unstated intermediate, such as 8 in (3 + 5) × 3, than incorrect alternatives, averaged over a fixed set of test problems.

Longer term: at these ‘developmental’ points, should they even exist, before, during, and after this workspace forms, we will treat the J-lens readout as an observable and calculate its susceptibilities (maybe using SGLD or something), and measure exactly which types of training data trigger the workspace to activate.

Something else could be to then rewind the model to just before the workspace forms, alter the training data to show it more of those specific "trigger" patterns, and see if we can force the workspace to form earlier. Would also be nice to do some fourier analysis here but this idea is not super well formed yet

probably the workspace doesn't form gradually; it "clicks" into place during a specific, short window of training like grokking. I predict this click is driven by data that requires holding unstated, intermediate facts in mind. Feeding the model more of this specific data should force the workspace to form earlier in training.

If the workspace just slowly gets better at the exact same rate the overall model gets better (following the standard training loss curve), then it probs wouldn’t work, and would just identify J-space as not being this super special thing thats a part of learning but something more gradual that just comes from the model getting slowly smarter.

Initially we would take a toy model and train it on something like addition and try to check that it forms a J space to handle extended reasoning. I would start with a small transformer, synthetic arithmetic data, and saved training checkpoints. If this works reliably, I would estimate which training-data patterns affect the representation and test those predictions by changing the data mixture. I would need GPU access and guidance on J-lens implementation and posterior sampling. If time is tight, I would drop susceptibility estimation first and prioritise a well-controlled developmental study across several training seeds.

Alice asks Bob and Charles the same question while privately producing a default answer. Bob and Charles independently answer or abstain, and are rewarded for agreement but heavily penalised for disagreement. Does uncertainty about the competitor make high-effort, non-manipulative answering the best strategy?

If the game-based protocol works, it would show that oversight can come from competition between capable but imperfectly aligned agents without requiring Alice to evaluate complex arguments herself. Instead, a simple game could use each agent’s uncertainty about its competitor to reward careful reasoning and make manipulation risky, while Alice only provides a trusted fallback answer. This could suggest a comparatively low-capability form of scalable oversight and further motivate studying competition as an alignment mechanism beyond standard debate. If it does not work, identifying why (e.g. excessive abstention, coordination on the same bad answer or incentives that fail to reward effort) would clarify the limits of oversight based on competition and which additional mechanisms/assumptions are necessary.

• [1]uses competition between two capable agents to help a weaker judge: the agents argue opposing positions, and the judge decides which is more convincing. It suggests that competition can make hard questions easier to oversee, but still relies on the judge following and evaluating an interactive debate.
• [2]studies a broader game where differently misaligned AIs strategically communicate with a human. It shows that competition can produce near-aligned outcomes under certain assumptions, but the incentive does not come from uncertainty about the other agent’s answer or from penalties for disagreement.

Additionally, GPT found [3], which I have not yet read, but which seems closely related. It studies agents who can choose whether to participate and exert costly effort, rewarding agreement and allowing penalties for disagreement. Under suitable payments, high-quality agents are incentivised to participate and exert effort. However, it is a peer-prediction/crowdsourcing setting rather than an AI-oversight setting, and does not include an aligned agent providing a default answer or the broader conversational protocol I want to study.

I would first formalise the simplest one-round version of the game. Alice, a weaker but highly aligned agent, receives a question Q and privately produces a fallback answer A. Bob and Charles are more capable but only mildly aligned. They see Q, but not A or each other’s response (B/C), and independently either submit an answer or abstain.

Here is a sketch of the game's rules. If both return A, Alice keeps A and they are rewarded. If both abstain, Alice keeps A and no reward is given. If one abstains while the other answers, or one gives A while the other gives a different answer, then Alice keeps A. If Bob and Charles independently give the same alternative answer, B = C ≠ A, Alice changes her answer to theirs and they are rewarded. If they both answer but disagree, B ≠ C, Alice keeps A and they receive a large disagreement penalty.

I would then analyse which strategies this payoff structure incentivises. In particular, I would ask whether uncertainty about the other agent’s performance makes careful, non-manipulative answering preferable to guessing, manipulation or abstention, and how this depends on the size of the disagreement penalty.

After understanding the one-round game, I would study simple extensions: finite quality scores V(x), repeated rounds forming a broader conversation in which later questions can follow from earlier answers or ask for supporting reasons, and finally simulations or LLM experiments measuring agreement, abstention, answer quality, and sensitivity to the penalty size.

I expect the protocol to work best when good answers are relatively constrained, while there are many more ways to manipulate or give low-quality answers. If Bob and Charles are also meaningfully different (e.g. in architecture, training or failure modes) then independent manipulation should be less likely to converge on exactly the same answer than careful reasoning toward a genuinely good one. I also think this structure may sometimes be created rather than simply assumed: by carefully defining what counts as an admissible answer, and what it means for two answers to be “the same” or “different”, one may turn a broad task into one where good answers concentrate more strongly than manipulative ones.

I would revise or drop the idea if, even after such careful task design, bad or manipulative answers remain just as likely to match as good ones, or if the protocol only works under very artificial definitions of agreement or unrealistically strong differences between the agents.

In three months, I would aim to produce a formal version of the one-round game, identify the parameter regimes in which careful answering, abstention or manipulation are preferred, and test the main predictions in simple simulations. This would include studying how effort, disagreement penalties and the relative likelihood of good versus manipulative answers matching affect the agents’ incentives. If time allows, I would then implement a small LLM experiment with two heterogeneous models or differently prompted agents, varying disagreement penalties, uncertainty and the definition of answer agreement. I would need basic game-theoretic analysis, a lightweight simulation environment and access to suitable LLMs. If the project becomes too large, I think I would first drop the multi-round conversational extension and then the LLM experiments.

In recursive LLM debates about synthetic experiment reports, does allowing one consistency check across branches reduce acceptance of false conclusions involving shifted definitions, compared with spending the same token budget checking a single branch, while maintaining acceptance of true conclusions?

An argument can contain individually plausible statements yet reach a false conclusion because a term changes meaning between them. For example, “accuracy” might mean performance on all examples in one branch and performance only on answered examples in another. Checking each statement separately could miss the substitution.

If a small consistency check catches these failures, it would suggest an affordable improvement to recursive debate: give judges access to the relationship between selected claims, alongside their local evidence.

If it fails, or works only by rejecting more truthful arguments, that would expose a limitation of this proposed fix. Either outcome would help determine how much context judges need before local checks provide useful evidence about an overall conclusion.

Irving et al. [1] introduced debate as a way for limited judges to assess claims beyond their ability to resolve directly. Its complexity-theoretic motivation relies on idealised players; it does not establish how reliably finite language models expose inconsistencies in natural-language arguments. AI safety via debate

Brown-Cohen et al. [2] address a related difficulty: dishonest debaters can construct arguments whose flaws are expensive to locate. Their prover–estimator protocol provides theoretical guarantees under specified assumptions, but those guarantees do not establish the effectiveness of a practical check for changing definitions. Avoiding Obfuscation with Prover-Estimator Debate

The closest motivation is Buhl et al. [3], §4.4.2. They explicitly identify inconsistent definitions across subclaims as a failure that judges inspecting only leaves might miss. They suggest combining features of cross-examination and prover–estimator debate, leaving the practical design unresolved. An alignment safety case sketch based on debate

Khan et al. [4] show that debate can improve judgements on reading-comprehension tasks, but this does not isolate consistency across recursive branches. Debating with More Persuasive LLMs Leads to More Truthful Answers

My proposed contribution is a controlled test of whether one limited consistency check provides a benefit beyond additional checking time.

I would first generate synthetic experiment reports from small tables computing the correct conclusions. Each report would support paired truthful and misleading arguments. Misleading arguments would change a denominator, dataset split, unit, or metric definition between branches. I would manually inspect the pilot to ensure that the intended inconsistency determines whether the conclusion follows.

I would implement a short recursive debate. A proponent commits to an argument tree; a critic selects claims to challenge. A separate judge receives the root question and limited evidence from the challenged argument. Modelsand prompts would be fixed before testing.

I would compare three conditions:

•Local checking: the judge sees the selected branch.

•Consistency checking: the critic can present two passages from different branches and explain their inconsistency.

• •Extra local checking: the critic receives the same additional budget but must spend it within the selected branch.

The latter two conditions would match total token allowances and excerpt lengths. All excerpts must come from the committed argument or report; the intervention receives no privileged correct definitions.

After the pilot, I would expand to approximately 500 reports, keeping development and test sets separate by report template. Thresholds would be set on development data to target 90% acceptance of truthful arguments. I would report both acceptance rates on held-out data, paired confidence intervals, and the full trade-off between rejecting false and accepting true claims.

Finally, I would let an attacker revise arguments against each condition using equal search budgets. This tests whether any benefit survives adaptation.

I predict that checking across branches will reduce false acceptance more than an equally expensive extra local check, particularly when individually correct statements use incompatible definitions.

I would treat a reduction of ten percentage points as a practically interesting target, while reporting uncertainty rather than declaring success from the point estimate alone. The benefit should survive an adaptive attacker and should not depend on substantially reducing acceptance of truthful arguments.

I would abandon this particular intervention if sufficiently precise results ruled out a useful improvement over extra local checking, or if its apparent benefit disappeared after controlling for truthful-argument acceptance.

I would also reconsider the benchmark if the baseline almost never accepted misleading arguments: that would mean it does not reproduce the proposed failure. A positive result would support a narrow protocol improvement, without establishing that recursive debate is generally reliable.

Three months would produce a reproducible benchmark, an implementation of the three conditions, an evaluation on held-out reports, and an analysis of representative failures.

Month one would cover report generation, protocol design, and piloting. Month two would run the main comparison. Month three would test adaptive attacks and write up results.

I would need API access to capable debater and judge models, a laptop for generation and analysis, and mentor feedback on the protocol. The pilot would determine the affordable evaluation size. If time became tight, I would drop additional model replications before dropping the matched-budget control or adaptive evaluation.

What is the right way to formalise the use of debate in an RL environment, and can debate be used to avoid rewarding undesirable behaviours in reinforcement learning?

If the answer to the question is yes, then we might be able to find a theoretically grounded way to mitigate reward hacking using debate, which would obviously be very significant for improving alignment. If the answer to the question is no, then I would expect it to highlight key limitations of debate as a paradigm, shedding new light on different classes of problems which are less or more difficult to resolve by debate and what they key features of these classes are.

I don't think there is existing theoretical work which has touched on this. The closest work is [2], an empirical study from Google Deepmind which shows that debate can be used to mitigate reward hacking in practice. The framework used by the authors was somewhat different to what exists in theory, and they observed other interesting phenomena which we would not necessarily consider from the start, such as B performing its own reward hacking. [1] is relevant for general intuitions about debate - how it can be “hacked” to give an unfair advantage to A and how this issue can be resolved - but it doesn't specifically deal with RL. [3] actively calls for solutions to this problems of this nature!

First, I would want to review the literature in this area more deeply to determine how others have thought about formalising related problems. It would also be beneficial to know which problems others are thinking about to guide my thinking.

I would first think about how exactly to formalise the idea of a debate where Bob is trying to detect reward hacking. I already have some ideas for this, which I can describe below:

Let $J$ be a judge which only approves elements of a language $L$ in $\{0,1\}^*$.

Suppose that A proves $x$ (for some $x \in L$) and supplies a reasoning trace $t$.

Then, B must try to identify a "subtrace" $s$ within $t$ which can be completed to prove $y$ for some $y \notin L$. Furthermore, this proof must use s in an essential way: that is to say, removing $s$ from B's proof should not be enough to prove $z$ for any $z \notin L$. The key here is that this implicates $s$ as an undesirable reasoning trace - it causally leads to proofs of undesirable elements of $\{0,1\}^*$.

Then, the question becomes which theorems follow across from existing paradigms - cross-examination, prover-verifier etc. I would hope to identify the complexity class of outputs from A that Bob can resolve in polynomial time and see how it compares to DEBATE. My approach would likely be self-adversarial - find the best thing that works for B, stress test it with pathological cases of A, repeat.

I think it may be difficult to carry across many of the existing theorems in debate because reward hacking is not necessarily easy to identify with local observations (which is necessary for CX). Therefore, B has a very challenging task to accomplish. However, through this process I would also expect to see ways to define obstructions to debate for reward hacking, which would be useful for gaining more insight on what causes reward hacking and which problems are most susceptible to it. We could potentially define classes in terms of how difficult it is for B to detect reward hacking.

I would hope to find mathematical results which shed more light on the classes which B can resolve with different time and space constraints. I think it should be possible within this time span to clearly identify the fault lines separating which types of reward hacking are simple to detect and which are more difficult and classify them, and identify "model organisms" of reward hacking within the theory. If it took more than three months, I would try to simplify the problem from detecting arbitrary failures of reasoning to something more narrowly scoped.

Can a computationally limited challenger find an invalid inference in a committed argument when differences from another valid proof need not indicate errors? I will study argument trees with public, fixed topology and locally checkable inference rules. Building on the paper’s plain-oracle model, I will investigate whether some families require inspecting a number of entries proportional to the argument’s length to find a violation with probability at least two-thirds, even with Merkle-authenticated access. This is a conjecture, not a consequence of the existing lower bound. I will then test whether locally checkable subtree summaries reduce inspections, while counting their added size and computational cost.

(A formal version is available at the end of this document)

Debate-based oversight relies on a challenger finding a flaw that a judge can check. A Merkle commitment binds Alice to her argument, but does not by itself guarantee efficient error search. A lower bound would identify settings where finding an invalid inference still requires inspecting much of the argument. A positive result would suggest argument formats that make this search cheaper, with explicit costs for additional structure.

Refereed delegation [1] gives efficient disputes over computations when an honest party can reproduce the correct execution. It does not directly resolve error search when no single reference argument is available. Prover-estimator debate [2] addresses obfuscated arguments through probability estimates, with stability assumptions for completeness; it does not establish the inspection cost of locating a violated inference in a committed proof.

First, I will construct a small family of formal argument trees with several valid proofs of the same claim. Each node will contain a claim, a rule identifier, and references to at most two premises. A judge will be able to check a named inference mechanically. I will specify the allowed rules, node sizes, trusted premises, and how the root represents the claim being defended. I will then define Bob’s access precisely: what he knows before questioning, which entries he can request, and whether he sees only their values or also Merkle digests and authentication paths. I will count preprocessing, inspected entries, communication, and computation separately.

Next, I will construct invalid arguments with isolated errors and derive lower bounds in the plain oracle model. I will investigate whether these bounds survive authenticated access, initially with an idealised random hash model. Any result relying on that model will be stated as such.

I will then test one restricted form of redundancy: additional claims about subtrees, accompanied by explicitly defined local consistency checks. These summaries will be treated as untrusted. I will seek either a search algorithm with a proved detection guarantee or an adversarial construction showing why the summaries fail.

I expect commitments alone to leave substantial search costs when an argument contains an isolated flaw and Bob has no useful reference proof. I expect improvements to require additional structure that helps locate errors, rather than repeated copies of the same claims. I would revise this prediction if authenticated summaries gave a sound sublinear search procedure under weaker assumptions than expected.

In three months, I aim to produce one precise access model, a proved lower bound or a restricted positive result, and small examples illustrating the assumptions.

We study the impact of debate/critique for a judge with limited-scope tool use. Concretely: Is the impact of tool use on the side of a judge bigger or smaller depending on whether the judge · 1. has no additional input, · 2. has a consultant, · 3. is presented with a debate? · We expect tool-use to improve the accuracy of the judge in all cases, but to be less beneficial in the debate format.

In prior work [1], the authors observed that the primary driver of debate's accuracy gains (in a reading comprehension task) was the tool which automatically verified quotes, rather than any verbal argument.

In our setup, we instead equip the judge with deliberate access to a limited-use verification tool, which answers the slightly different question of whether tool use trumps debate in verifiable tasks.

If tool+debate beats tool-only, its value lies in directing a weak judge's limited checks to the right place.

If tool+debate does not beat tool-only, much of debate's measured benefit in prior work may simply be due to verification access.

In [1] and [2], all debates were performed with verified quotes, obfuscating the cause of the accuracy gains. Disentangling debate and verification tools would shed light on this.

The authors of [1] note that [2]'s results suggest verification rather than argument drove accuracy, and list judge code execution as future work.

Elasky et al. [3] study proposer–critic debate on verifiable code and logic. They find a single critique recovers most of debate's benefit, and that debate helps only when the judge checks the critic's claims rather than summarising them.

PROClaim [4] adds live retrieval to a courtroom-style debate, with the presiding judge refining the debaters' search queries. Retrieval drove most of its gain over standard debate, and a single model with retrieval alone scored higher than the full pipeline. Its judges are as strong as its debaters, and nothing is optimized against them. Training work [5] reports that hacking the judge is the default outcome unless the players are restricted.

Step 1: We use coding tasks (e.g. from LiveCodeBench) released after the models' training cutoffs, with private tests as ground truth. For each problem, generate one correct solution and one incorrect solution that passes the public examples but fails private ones.

Step 2: A weak LLM-judge with a Python sandbox (limited number of runs) picks the correct solution under the following conditions:

no tool,

Python sandbox (tool-only),

sandbox plus one consultant (a stronger model such, randomly assigned to either solution, word-limited response),

sandbox plus one-round simultaneous debate, same word limit per debater, with debaters allowed to propose test inputs,

debate without the sandbox.

Measure accuracy and how often the judge endorses the wrong solution. The judge prompt, tool budget are fixed.

Step 3: Ablations. Reuse the prompts of debate transcripts, but mask test-input proposals/written arguments as comparisons. Perform the same accuracy measurements.

Step 4: Vary the strength of the judge.

Tool-only will beat no-tool significantly, because many wrong solutions fail on simple edge cases. One-round debate will further improve accuracy over tool-only for the judge, mostly by cutting false positives: the judge runs test-cases suggested by the debater that catch the mistake. Consultancy will not beat tool-only, and will do worse when the consultant argues for the wrong solution. The debate gain should shrink as the judge gets stronger or the tool budget grows, and tool-gain is smaller in debate.

Three months would produce:

the dataset with private-test ground truth,

all five conditions for the judge, with confidence intervals,

both mechanism ablations,

a short write-up.

It would need a sandboxed execution environment, API budget, and one GPU for the open-weight judges. If time runs short, I would drop the extra judge sizes and tool budgets first, then the no-sandbox debate condition.

In Heckett's debate model (claims arranged in a tree; neighbouring claims joined by links, i.e. the judge's belief about how strongly two claims agree; all links very strong), do far-away claims lose the power to swing the verdict exactly when μ < 1, μ being the average number of further claims a small change in one claim passes on to?

In Heckett's model [1], claims are yes/no statements. Links encode how strongly the judge thinks pairs of claims agree or disagree; claims and links form a factor graph, here a tree (no cycles). Two debaters alternately reveal claims, and the judge updates on what is revealed. Example: question q links to claim a (no evidence of its own), and a links to b, known true. If links make agreement 7.4 times likelier than disagreement, revealing b moves q's log-odds by 1.33; at 1.65 times, by only 0.12.

Debate is local when claims near the question (a few links away) settle it, and non-local when a far-away claim can still flip the verdict however much has been shown. Heckett argues the two regimes need different protocols and names non-locality as a next direction [1]; where it begins is unknown.

If yes, one computable number, μ, says whether a longer debate drives the error to zero, and how much independent evidence restores locality. If locality breaks earlier, physics stability calculations overstate where unstructured debate can be trusted.

Setup [1]: a tree, each claim linked to d others; link strengths are random with typical size s; each claim also has its own random evidence of typical size κ·s (κ = 0 is Heckett's setting). A core is a connected set of claims around the question that a debater can reveal in h moves; its oscillation is the largest change in the verdict that revealing claims outside it could still cause. Debate error is at most the oscillation of any core that fits the budget [1]. For balls (all claims within r links), the AI-generated companion [2] proves average error $\le C\,h^{-\gamma}$ when ρ(d, s) < 1, ρ being a cruder, larger stand-in for μ. For very strong links ρ → 0.67, 0.89, 1.07, 1.23 for d = 3…6, so from d = 5 the proof is silent and "has not established nonlocality" [2].

Very strong links make this a known physics problem: claims are spins, links are couplings, evidence is local fields, and the judge's update is the zero-temperature spin-glass update on a tree. A small change crosses each link fully or not at all, spreading like a branching process with mean μ. μ = 1 in the (d, κ) plane is the zero-temperature de Almeida–Thouless line ("AT line") [3, 4]. Open: whether debate stops being local exactly there (oscillation is a worst case; worst-case and typical thresholds can differ on trees [5]); whether the change is gradual or abrupt (the related random-field Ising model jumps abruptly [6]); and whether best-chosen cores behave like balls (they leave ~10× less oscillation [2]).

The precise claim, for very strong links: the oscillation of balls, and of the best-chosen cores, shrinks to zero as the radius r grows (for almost every random draw of the links) if and only if μ < 1. Here μ = (d−1)λ: each claim passes a change on through d−1 further links, crossing each with probability roughly λ (formally, λ comes from linearising the update [3]).

• 1.Exact computation (week 1). A claim's log-odds never decreases when a positively linked neighbour's log-odds increases (never increases, for a negative link). So passing [lowest, highest] log-odds intervals up the tree gives the extreme verdicts over all unrevealed claims [7]. I compute this on a large pool of sampled intervals (population dynamics [8]) and on explicit trees of 10⁶ claims, 200 random draws per (d, κ). Check: at d = 3 the widths must shrink by at least a factor μ ≈ 0.58 per level.
• 2.Check the preliminary numbers. A preliminary computation (one random seed) reproduces the constants of the companion's error bound (C = 14.58, γ = 0.901) and the known AT-line field 0.358 for a uniform field on a tree with d = 3 [3]. It gives μ = 0.88, 0.99, 1.10 for d = 5, 6, 7 at κ = 0. Repeat with at least 5 seeds and larger pools; d = 6 is borderline.
• 3.Gradual or abrupt change from local to non-local? More evidence pins claims, so larger κ lowers μ: the AT line sits near κ ≈ 1.5 at d = 7 and κ ≈ 2.4 at d = 8. Across it, and at d = 6, start the computation from the widest possible intervals and from intervals 100× narrower. Gradual: both shrink to zero below the AT line and both stay wide above it. Abrupt: just below the AT line, the wide start stays wide while the narrow one shrinks.
• 4.Bound vs actual debate. Near the AT line, compute the best-core oscillation as in [2, §7], and the exact error of optimally played debates with up to h = 4 moves.
• 5.Proofs for very strong links. Below the AT line, adapt the companion's Lemma 5.1 with μ in place of ρ; above it, show that wide intervals persist.

Working hypothesis: a gradual change exactly at the AT line; the abrupt jump in [6] is the main reason for doubt. Below the AT line, the width (the range of verdicts still possible) should shrink by a factor μ per level, so the error falls like $h^{-\log(1/\mu)/\log(d-1)}$. That is the companion's exponent $\gamma = \log(1/\rho)/\log(d-1)$ with the smaller μ in place of ρ. The preliminary computation agrees: at d = 5 the width shrinks by about 0.87 per level (μ ≈ 0.88). At d = 7 it stays near 6.2·s in all 20,000 samples, and even a 100× narrower start grows back to that width.

What would change my mind:

• •A range of κ just below the AT line, robust to seeds and pool size, where wide starts stay wide and narrow ones shrink. Then the change is abrupt, and debate stops being local before the AT line.
• •Widths shrinking above the AT line (μ > 1). Then the claim is false.
• •Best-core oscillation still shrinking above the AT line. Then μ = 1 limits the companion's bound, not debate itself.

• •Month 1: exact computation, multi-seed check, and a map over (d, κ) from both starting widths.
• •Month 2: the gradual-vs-abrupt test with seed and pool-size scaling; bound vs actual debate for small h.
• •Month 3: proofs for very strong links that debate is local below the AT line (likely) and non-local above it (harder), plus a LessWrong write-up, ideally checked with Heckett.

CPU only, minutes per (d, κ) setting. Drop first: links of moderate strength, where the boundary becomes a new curve $s_c(d, \kappa)$. Then the exact-debate computation. Then the non-locality proof.

LLC estimation of sparse autoencoders when learning from activation space of different layers of an LLM

LLC estimation only works for simple models and is only well defined when you reach local minima of the loss landscape. This project attempts to study degeneracies in learned representations within transformers (arguably more relevant for alignment) and draw some conclusions on what is easier or harder to be learned.

Not clear if any work answers this.

Prior work distinguishes architectural symmetries, sparse-code non-identifiability, and empirical instability. SAEs trained on identical data can learn substantially different features across seeds [1]. Yet unstable features can collectively occupy reproducible low-dimensional subspaces, suggesting ambiguity in individual atoms rather than entirely different representations [2]. Classical dictionary-learning theory gives conditional uniqueness guarantees requiring sparsity, dictionary conditioning, and sufficient data diversity [3]. Recent iSAE work proposes conditions and regularization for improved identifiability, although our audit identified a proof issue requiring clarification [4]. Feature-targeted learning-coefficient experiments study SAE directions through the underlying language model, not SAE parameter degeneracy [5]. These distinctions matter for research.

• 1.Use frozen Pythia-160M sixth-layer MLP activations, following [1].
• •Train three SAEs with identical data, order, architecture, and hyperparameters, varying initialization.
• •Reproduce encoder/decoder feature matching and shared-feature fractions.
• •Record reconstruction error, firing frequencies, dead features, and sparsity.
• •Hold out complete sequences for evaluation.

Start with a smaller dictionary + token budget rather than 32K-latent, 8B-token

experiment from the paper..

• 2.Construct small synthetic activation datasets and SAE configurations with:
• 1.Duplicated decoder atoms: known compensating code freedoms.
• 2.Well-conditioned sparse factors with diverse supports: an identification-friendly control.
• 3.Restricted support coverage or strongly correlated factors: controlled weak constraints.

I expect to reproduce substantial seed dependence while finding that reconstruction quality varies less than individual feature identity.

Some unmatched features should nevertheless span similar subspaces, consistent with existing work. The key question is whether these alternatives reflect locally connected reconstruction freedoms, isolated solutions, or incomplete optimization.

In synthetic controls, duplicated atoms should produce exact compensating directions, whereas well separated, sufficiently observed factors should be more strongly constrained. Correlated or rarely sampled factors should introduce weaker constraints, although correlation alone need not imply exact nonidentifiability. These controls should establish which degeneracy measurements are reliable before interpreting transformer results.

• 1.Separate coverage from optimization. Compare repeated training on the same activations, random additional data, and targeted additions containing underrepresented activation patterns.
• 2.Test architecture dependence. Compare TopK and ReLU SAEs at comparable reconstruction and sparsity. Persistent patterns across architectures provide stronger evidence about the transformer.
• 3.Connect local and cross-seed geometry. Test low-loss paths between aligned solutions; local degeneracy alone cannot establish connectivity.
• 4.Expand cautiously. Add one transformer layer or checkpoint before attempting large mo

In deep linear networks trained to zero loss on a task with many interpolating solutions, does Muon converge to minima with different local learning coefficients than stochastic gradient descent and how does better generalization correlate with the local learning coefficient?

The most recent frontier models, like GLM 4.5/5, KimiK2/K3 and DeepSeek-V4 are trained with (some version of) the Muon optimizer and it is getting more traction [1]. Therefore, understanding if the trajectory a model takes in its loss landscape affects a model’s generalizability seems important.

Studies have already shown that Muon converges faster than other techniques [2,3] and that grokking happens earlier [4]. Using a Muon optimizer in deep linear networks (DLNs) shows that singular values are learned simultaneously and small singular values converge to optimality before large ones [5, 6]. This order is the opposite of the saddle-to-saddle dynamics of gradient descent (GD), where the time-scale of learning is inversely proportional to the learned singular value [7].

Understanding the regions of the landscape different optimizers reach seems important to alignment because it informs us on how well a model generalizes [8]. Learning that models generalize differently using a metric different than testing accuracy may reveal more insight. Singular learning theory (SLT) uses the local learning coefficient (LLC) to study the geometry of the loss landscape [9] with results showing the LLC can reveal when a transformer learns a particular circuit, improving its generalizability [10,11]. SLT compares learning with stochastic gradient descent (SGD) to generalization in Bayesian posteriors [12] and would benefit from extending their SGD-Bayes correspondence to the Muon optimizer, informing on the usefulness of SLT for studying frontier models.

I would start by reproducing the results in [5], which trains a DLN on 14 pairs of 7x7 input/output sources. I would train the model with SGD, spectral GD and Muon (momentum and Newton-Schulz), with different seeds (maybe use a learning rate schedule). After each epoch of training, compute the out-of-distribution (OOD) error on the 35 unseen examples, compute the rank of the Jacobian to get a lower bound on the LLC. Do an LLC estimate with stochastic gradient Langevin dynamics (SGLD) with a sweep over hyperparameters using the devinterp library.

If there is a robust gap in generalization accuracy between different optimizers, then I can study this mechanism in other types of tasks. For example, I could repeat step 1 on matrix factorization tasks like presented in [13] where gradient-based optimization is known to implicitly produce low-rank solutions, leading to better generalization.

Calculate the LLC analytically after every training epoch using a similar process as presented in [14] to establish a mechanism explaining why different gradient-based optimizers induce different generalization properties as studied in [15, 16].

Recent work [15, 16] shows that Muon achieves better generalization. My hypothesis is that this will show up in the LLC, where we’ll see a lower LLC (lower complexity) when the generalization is better.

If the LLC cannot tell the difference between different generalization scores for different optimizers in step 1, then I would drop the question. If step 2 does not show that the observed phenomenon happens for a variety of tasks, I would also stop. I would completely drop the question if I need to do some algebraic geometry to determine the mechanism in step 3 (or find a mathematician that knows how do to it).

The first month would probably encompass steps 1 and 2. Finding the mechanism comes next and is most probably the hardest task. If this turned out to be a three year project, then I might cut trying to get analytical results and would instead find principled approximations. Maybe there are scaling laws that could inform us on the dynamics of the loss near the minima based on the Bayesian learning results?

Can Kronecker-factorized / adaptive preconditioners (RMSProp, Shampoo, SOAP, SOAP-T), used as the metric of a Langevin sampler with matched noise, give faster-mixing and more accurate local learning coefficient (LLC) estimates than plain SGLD — and does an exactly corrected sampler (preconditioned MALA) confirm that the remaining error can only be attributed to mixing, while the bias remains controlled?

The LLC is the workhorse of developmental interpretability, and scaling its estimation could enable better midtraining intervention approaches. Besides, as LLMs are already pretrained with adaptive optimizers, reusing preconditioner estimates throughout training could be viable. Every estimate today rests on SGLD hyperparameter choices (step size, localization, chain length), but show high variance and slow convergence, which makes them computationally prohibitive for midtraining interventions at scale.

The local learning coefficient in SLT is usually estimated with localized SGLD-type samplers that display high heteroskedasticity [7] and are validated on deep linear networks with closed-form λ [8]. SGLD samples a tempered posterior with isotropic noise [3]; pSGLD adds an RMSProp-style diagonal metric and scales the noise to match, preserving the target up to a curvature term [4]. Richer metrics have entered inference in the same spirit—noisy natural gradient treats Kronecker-factored curvature as a variational covariance [5] — and Shampoo/SOAP-type preconditioners are now framed as structured Fisher approximations [6].

Experiments, small scale first.

(1) Teacher–student DLNs (3–4 layers, 10²–10³ parameters, rank-r optimal solution with every factor of rank r) where λ is exactly known.

(2) Samplers: SGLD in the metric P⁻¹(nβ∇L + ∇prior) with noise $\sqrt{\mathrm{lr}}\cdot P^{-1/2}\xi$, for P ∈ {RMSProp, Shampoo, SOAP, SOAP-T};

(3) Full-batch preconditioned MALA as the exact reference, adaptation frozen after burn-in. (4) Sweep step size × localization γ at a fixed step budget; report (λ̂−λ)/λ, chain spread, acceptance, divergence rate; compare against long-chain identity SGLD/MALA.

(5*) Use learnt diffusion-type sampler [1] on DLNs or estimated learning coefficients in tractable networks (tensor networks display non-trivial degeneracies as well – even bilinear/multilinear might show us something), and estimate generalization to higher-dimensional spaces. However, it still relies on some ground truth and transformer-type models to generalize. Potentially, as the mapping seems to be permutation invariant (check!), we don’t have to care about length generalization that much.

A plot of relative LLC error vs. step size per metric, at fixed compute, in which at least one Kronecker metric reaches λ within ~5% where identity SGLD is still 30–70% low; MALA acceptance staying high (>0.5) at the step sizes where that happens. A clean negative result — no metric beats identity at equal budget — is also useful, as is the already-established collapse of gradient-only preconditioning.

Second priority: try to derive analytical LLCs for bilinear and tree tensor networks (better with shared factors), as they’d allow to get the proxy for the precision of estimates for real networks.

Potential extensions:

• •Trust region approaches to control for the e.g. spectral norm of a solution. We might impose some priors on the space of matrix parameters with prespecified norm $\|\cdot\|_{1\to\infty}$ or $\|\cdot\|_{2\to 2}$, and then define priors over it. Overall, optimizers during the learning process tend to impose their inductive biases on the set of solution matrices. The trust region allows us to account for it.
• •Low-rank fisher approximation extensions [2] could increase the efficiency if initial approaches prove to be scalable, however matching the checkpoint preconditioners is already enough for devinterp.

Does incomplete coverage of reachable belief states create additional low-loss parameter freedoms in a sequence model? Are those freedoms reduced by observing more diverse histories rather than simply training longer?

You Are What You Eat [1] argues that understanding how data shapes internal structure and generalization is central to reliable AI. We investigate this relationship in HMM sequence prediction, where the correct predictive beliefs are known. By independently varying history coverage and optimization time, we test whether poorly represented belief distinctions reflect missing statistical constraints or incomplete learning, and whether local loss geometry helps distinguish these explanations.

Training scale vs. training data scope and belief state coverage

I want to investigate deeper the role that training data plays in affecting belief states of the trained model, rather than assuming that the training data has a nice distribution, it would be interesting to consider the training data itself as a parameter we could adjust and optimize for maximally approximating some interpretation of ‘the real world’. A model may represent belief states poorly due to not being sufficiently optimized, because the training histories do not constrain relevant distinctions or because its architecture cannot express the ideal representation. We want to understand how data constraints the belief states and which changes in the learned model are still weakly constrained.

• [1]You Are What You Eat – AI Alignment Requires Understanding How Data Shapes Structure and Generalisation.
• [2]Transformers Represent Belief State Geometry: small transformers can linearly encode HMM beliefs, including distinctions beyond immediate next-token predictions.
• [3]Constrained Belief Updates: architectural constraints help explain the particular belief-update computations and geometries transformers implement.
• [4]Structure and Scale: tracks belief-representation refinement with training, but increases optimization and data exposure together.
• [5]Refined Local Learning Coefficients: provides a framework for studying loss geometry relative to particular components and input distributions.

• 1.Reproduce the starting phenomenon in [2] for the scaling of belief reconstruction errors for a small transformer trained on Mess3 HMM
• 2.Train longer on unchanged finite corpus (training dataset), increase corpus size while controlling optimization steps, compare equal-sized corpora with different coverage of reachable belief state
• 3.Connect coverage to degeneracy by perturbing specific small parameter blocks at selected checkpoints. Measure whether perturbations that barely change predictions alter belief representations. (check the following: $D_P(\theta) = E_{s\sim P}\,\mathrm{KL}(P_{\theta_0}(\cdot|s)\,\|\,P_{\theta_0}(\cdot|s))$, aka changes from a reference model). Note that we need to use finite perturbations and not just Hessian null directions.

Result: Evidence that particularly poorly covered belief distinctions correspond to parameter freedoms that affect held-out prediction - better coverage constrains them more effectively than additional optimization alone.

Plot: belief-reconstruction error + held-out sensitivity across independently varied data coverage + training time

Negative finding: representational refinement is explained by optimization / architectural constraints with no additional explanatory benefit from the degeneracy measurements

Fine-tuning language models with parameter noise reduces cryptographic backdoor attack success rate, while maintaining model accuracy. Standard fine-tuning fails to remove the backdoor.

Training LLMs is computationally intensive, driving widespread adoption of open pre-trained models that might contain backdoors. In particular, these backdoors can be exceptionally hard to detect.

A positive answer will provide a practical preventive safety measure to mitigate backdoors without needing to detect them or compromise model accuracy.

A negative answer means that the backdoor cannot be broken without compromising the model’s accuracy. This in turn suggests that we must explore alternative sanitization techniques, or shift toward zero-knowledge proof frameworks that cryptographically guarantee no unauthorized post-training modification of shared models has occurred.

Standard fine-tuning is widely considered an effective backdoor mitigation technique because it induces catastrophic forgetting of backdoors [1]. Training in one epoch or even in few batches already mitigates common backdoors. However, in [2] it is showed that cryptographic transformer circuits can implement fine-tuning-resistant backdoors. By encoding logical circuits with zero-gradient ReLU gates inside the transformer, malicious behavior only activates under specific trigger inputs. During fine-tuning, zero gradients prevent backdoor weights from being updated. In [2], the authors acknowledge that these backdoors are sensitive to parameter noise, but suggest that attackers could implement error-correction subcircuits to increase backdoor robustness. While weight-noise injection has been studied as a defense against standard backdoors [3, 4], several fundamental questions remain open. First is the empirical evaluation of the trade-off between parameter noise intensity and error-correction circuit capacity in LLMs. Second is whether dynamic parameter noise injected during fine-tuning can reactivate zero-gradient ReLU gates to force gradient updates without destroying core language utility. Third, it remains unknown whether noise-injected fine-tuning can serve as a universal, trigger-agnostic safeguard to sanitize open-weight models against robust cryptographic backdoors.

Following [2] we have to build cryptographic backdoors with different levels of error-correction subcircuits. Next, we should evaluate the baseline performance of the models as well as their Attack Success Rate when using trigger inputs.

Then, fine-tune the models on clean instruction data using three different strategies: standard fine-tuning, static noise added to model weights and dynamic noise injected into weights during each training step. The range of noise intensity should also be a parameter and we should repeat the experiments using different values of it.

After evaluating performance with respect to the baseline we should be able to find an optimal noise level that leverages accuracy and attack success rate. Special attention should be given to the backdoor weights and see if fine-tuning corrupts the cryptographic circuits.

We expect that the dynamic parameter noise strategy will corrupt the thresholds required for the cryptographic backdoors. A good result would be having an attack success rate below 5% while preserving accuracy within 1% of baseline performance.

Conversely, I would drop this approach under two conditions. First, if error-correction subcircuits work successfully, keeping the backdoor active (ASR > 20%) while blocking gradient flow through zero-gradient gates. Second, if the noise scale required to break the backdoor simultaneously causes catastrophic utility loss (>5% drop in clean task accuracy). Either outcome would prove that weight-level noise cannot selectively sanitize precision-sensitive cryptographic logic without destroying the model's general language capabilities.

For a three month project we would aim to have a benchmark of backdoored language models with varying error-correction capacities. An evaluation on how different noise-injected fine-tuning techniques impact attack success rate and accuracy performance.

This study could potentially be extended to several model sizes. We would also need to have access to a clean instruction dataset.

If time constraints go beyond three months, we would prioritize studying backdoored models with fewer error-correcting subcircuits. Also, we would consider lower model sizes. We believe that the behaviour observed in small models might scale with bigger models.

In the case of a teacher-student pair that is initially identical (MLP with two layers trained on MNIST) but distillation is done only with trigger-free samples, is the prediction of the student’s margin on trigger inputs by the lazy NTK regime consistent with the activation of the teacher’s backdoor?

Nowadays, distillation from models we don’t fully trust is a routine, and subliminal transfer implies filtering the distillation data does not remove what the teacher carries. In particular, backdoors are one of the worst things that can be transferred.

If it holds, we get a computable criterion for when a conditional behaviour will be inherited. We would know which distillation sets are safe and it would prioritise monitoring the projection of the teacher’s update onto the span of the distillation gradients.

If it fails, then continuous-trait kernel accounts of subliminal learning do not extend to thresholded behaviours (the safety-relevant case). If it holds only in the lazy regime, then we know that the sharp vs. continuous thresholds seen empirically in LLMs need a feature-learning explanation, so we know where to direct our efforts.

From what I have found:

• [1]prove that a small gradient step on a teacher’s outputs move sthe student toward the teacher in parameter space and show that transfer vanishes with different init. It does not say anything about i fa behaviour crosses an activation threshold. [2] studies multistep transfer empirically on MNIST. [3] give a theoretical kernel framework bu treats continuous traits. A backdoor is a dormant function with a threshold, and it is either active on $x^*$ or it is not.

Empirically, we know that backdoors are known to transfer: [4] uses backdoor as the trait and finds a sharp threshold in Llama2 and a continuous in Qwen2.5, but with no predictive account on the difference. We also know that poisoning survives paraphrasing the whole dataset and includes password-triggered behaviours [5], and unsafe behaviours transfer in agent distillation [6].

None of these directions involves a written condition for when a trigger-conditioned behaviour crosses its activation threshold in the student.

Setup. Train a wide two-layer MLP on MNIST. Finetune the teacher network, using the same weights as before, by adding the backdoor – a small patch that makes it predict class t.

Prediction. Calculate the NTK at initialization for the distillation set (consisting of noise and clean digits, but without the patch), and for patched images. Use it to predict whether the student network's margin on patched images is going to be positive [8, 9].

Measuring. Distill the student network from the same initialization on that dataset and compare the predicted margin with the measured one and the ASR.

Breaking the theory. Reduce the width of the network and increase the learning rate in order to escape the lazy regime and observe where the predictions stop holding.

Varying data and backdoor. Change the distillation data in such a way that the amount of teacher's update lying in the span of the distillation gradients can be varied, and check if the ASR correlates with that projection. Vary the strength of the backdoor in order to observe the threshold of its switching on. Control: another initialization shouldn't contain the backdoor.

If time allows, repeat with cross-entropy instead of MSE, or on a small language model with LoRA.

I expect to see that the formula correctly predicts the backdoors transfer in the case of wide networks trained with a small learning rate. And that as the network leaves the lazy regime, the prediction gets worse and transfer becomes abrupt and unpredictable. ASR should grow with the projection of the teacher's update onto the span of the distillation gradients.

I would change my mind if the formula starts failing even in the regime where it should hold (wide network, small learning rate). If the backdoor never transfers in this setup I would change the setup.

Month 1: pipeline and the lazy regime test.

Month 2: regime sweeps and data/projection experiments

Month 3: The cross-entropy variant, a write up and a clean statement of the condition and where it fails.

Drop first: The LLM/LoRA extension, then the cross-entropy variant

I would like to devise a reference model-free backdoor detector. Concretely, I want to focus on a ResNet18, fine-tuned on CIFAR-10, backdoored a la Eggen et al.. My goal is to contradict their conjectured white-box undetectability. Can spectral methods help us achieve this?

There are several open-weight model providers across the internet. Assume you are a business that asks one of these providers to train or fine-tune a model. Imagine an adversary handed you a backdoored model. Imagine to have a classifier that in principle works perfectly, except when a trigger is added to the input. For example, a ship image plus the trigger is classified as a truck. This would be bad if you asked your model to buy you a truck.

It would be nice to elicit backdoor behaviour. However, given this could be difficult, we would be happy even by detecting the presence of a backdoor. If we can answer this question, we, as people or as a customer, would be able to, hopefully robustly, verify a model handed to us before deployment.

To make this concrete, I decided to work on this topic in the context of the Eggen et al. paper as it has a handful of specific parameters we can play with.

Generally, for untrained models, this is a very hard problem. However, for trained ones, the question is still kind of open.

We saw in class that Goldwasser et al. [2] prove white-box undetectability for a random-ReLU construction under a sparse-PCA hardness assumption. This proof does not cover Eggen’s trained networks [1], where undetectability remains conjectured.

I will attack the problem from a more practical view. My question is therefore whether a reference-free test, e.g. spectral test, detects Eggen’s published configuration while the attack remains useful. Ideally, I would identify a detectable regime, but not rule out regimes given it is a practical project.

I expect that at their published θ = 0.1, a reference-free spectral test on the inserted layer's weights separates backdoored from clean models with AUC above 0.90. Then I vary the parameters until detection fails or the backdoor is useless.

Reimplement the backdoor of [1] on ResNet18 with CIFAR-10 and match their attack success and clean accuracy.

Train about 20 clean models with the inserted layer, to see how much a normal trained layer varies. Measure its weight variance σ², which the paper does not report.

Each backdoored model has a clean twin that differs in one layer. The difference should give back the planted direction. This checks my code and is the best case for any detector.

Using only the weights W, compute the largest eigenvalue of W Wᵀ / d, the variance of each row, and the variance of W projected on a class direction I estimate. Compare with the clean seeds. Run DFBScanner [5] and HTell [6] too.

Vary θ and sparsity. Record the attack gain over the clean model and the AUC of the best statistic.

Compare where detection fails with the sparse-PCA thresholds [9], using σ².

My prediction is that at high θ it is easy to tell the backdoored spectrum from the benign one, and that the published θ = 0.1 is already high. If the inserted layer stays close to its initialisation, θ = 0.1 is about say 150 times the weight variance, so I expect one eigenvalue far outside what the clean seeds show, and an AUC above say 0.90.

Going down in θ, I expect detection to hold as long as the backdoor still helps the attack. The trigger alone already fools the clean model most of the time, so the backdoor has little room to help at small θ.

I would drop the question in two cases:

At θ = 0.1 the clean seeds show eigenvalues as large as the backdoored ones, so the AUC stays near 0.5. Then their conjecture has its first evidence, and I would write that up.

I cannot reproduce their attack success and clean accuracy, so I am not testing their construction.

Given I did not find code of the original paper, the first month would be devoted to

reproducing the main paper results

make a baseline

ideally, I get in touch with someone really good at math to provide a proof or not of the paper’s conjecture

The second month, and first half of the the third month

start from the basic paper’s setup for theta = 0.1 and use spectral methods to detect the backdoor. Produce first results.

the sweep over parameters would produce space where conjecture is false

Rest of the time in writing notes by hand, no AI

I would need access to Runpod GPU, say 300-500 hours of an A100

We train small transformers on Mess3 with a backdoor i.e a fixed 4-token trigger that switches the tokens to a second HMM. We want to find out whether, on normal inputs where the trigger never finishes, the residual stream still tracks how much of the trigger has been seen, whether an attacker can remove this without making the backdoor less reliable, and whether the same check finds a planted tri

• •Often when checking a model for a backdoor we don’t know what the trigger is.
• •Before the trigger is finished, how much of it has appeared doesn't change the model's next prediction at all, so there’s no obvious reason for the model to keep track of it.
• •It does change what could happen a few tokens later though, and Shai et al. [1] found that transformers store information about the whole future, not just the next token. We aim to investigate whether normal training forces a backdoor to leave a trace, or whether a model can hide one the way the hand-built backdoors in [4] and [5] do.
• •If the trace is there and hiding it costs reliability i.e defenders get a way to look for backdoors without knowing the trigger, and attackers face a trade-off we can measure
• •If an attacker can hide it for free i.e learned backdoors can be as hidden as the hand-built ones, and probe-based detection like [2] can't be trusted against an attacker who knows about it
• •Known answer i.e in the toy process we can work out exactly what the model should be tracking, so we can check what a detector actu

• •Shai et al. [1] showed that transformers trained on data from a hidden Markov model (HMM) store their beliefs about the hidden state as a geometric shape in the residual stream, and that this shape holds information about the future beyond the next token. Their processes had no rare events though, so every part of the belief mattered for prediction all the time
• •Hubinger et al. [3] showed that backdoors trained into LLMs survive safety training
• •MacDiarmid et al. [2] then showed that simple linear probes on those models can tell when the backdoor is about to go off, using generic yes/no prompts that don't mention the trigger
• •But this is on inputs that contain the trigger, the attacker wasn't trying to dodge the probe, and we don't know what an LLM is really tracking inside, so it's unclear what the probe is picking up
• •Goldwasser et al. [4] and Draguns et al. [5] build backdoors by hand that are provably hard to detect, by hiding the trigger inside cryptography. This shows detection can fail by design, but says nothing about backdoors a model learns in training, or what hiding one costs
• •Piotrowski et al. [6] covers how one layer computes the belief update on Mess3, but only for the normal process with no backdoor
• •So no one has checked whether a learned backdoor shows up on normal inputs, whether an attacker can train it away, or whether a detector tested where we know the answer works on a real LM

• •Toy experiment on finding trace (Pt1)
• •We can build the backdoored process as one HMM. The hidden state is the normal Mess3 state plus how many trigger tokens have been seen in a row so far (0 to 3). Once the trigger is complete, the tokens come from a second, different Mess3 for the rest of the sequence
• •The code from [1] gives the exact belief for every input
• •Train small transformers with the setup from [1] + model trained on normal Mess3. First check the loss is close to the best possible, both overall and right after the trigger, so we know the backdoor was learned
• •Then, on normal inputs only:
• •Run a direct check at each layer, train a linear probe to read off trigger progress. Progress only depends on the last 3 tokens, so the control model shows how much you can read for free from recent tokens
• •Blind check (what a defender could do without knowing the trigger) i.e train a probe for the normal Mess3 belief only, then group what it can't explain and see if the groups match trigger progress
• •Plot the main directions of variation (PCA). Here in theory we predict a separate copy of the Mess3 shape for each progress value
• •Vary trigger length (3 to 6) and number of layers to see when the trace appears / disappears
• •Can an attacker hide it? Toy + theory (Pt 2)
• •Theoretically we want to prove two normal inputs that differ only in trigger progress lead to different futures, so any full record of the belief has to include progress (this part is close to the definition). How different those futures are is limited by how likely the trigger is to finish soon, seems like about 1 in $3^{L-k}$ for progress $k$.
• •Attacker 1 uses triggers with no step-by-step structure i.e a random set of 4-token strings with the same total rate. Same checks as Part 1
• •Attacker 2, training against the probe i.e we train a probe for trigger progress at the same time as the model, and penalise the model whenever the probe succeeds (gradient reversal, as in [7])
• •For each attacker, plot how big the trace is against how reliably the backdoor still works.
• •Fine-tune on normal Mess3, like the safety training in [3], and check whether the backdoor and the trace disapp

• •Toy model prediction: I believe trigger progress is much easier to read in the backdoored model than in the control, the PCA plot shows a separate copy of the Mess3 shape for each progress value + blind check matches progress well above chance
• •Attacker prediction: the trace gets fainter as the trigger gets rarer, roughly as we claim, and both attackers can shrink it but only by making the backdoor less reliable
• •Real model example: the blind check picks out the trigger well above chance against the plain backdoor, and does worse against the attacker
• •What would change my mind:
• •The backdoor works fine but trigger progress is no easier to read than in the control. Then the model only checks for the trigger at the moment it finishes, the belief geometry result doesn't cover things the next prediction doesn't need so the attacker/ real model ideas would fall through
• •An attacker removes the trace with no loss in reliability. This means the trace isn't forced, and learned backdoors can be as hidden as “hand built” ones
• •The check works on the toy but finds nothing in Pythia. Then the toy result doesn't carry over, main work here would be to find which of our assumptions failed/ fell through

• •Month 1: Part 1 with the 4-token trigger
• •Month 2: Part 2, both attackers, trace vs reliability plots
• •Month 3: Part 3 on Pythia-160M.
• •If it ran over the time, we can drop the trigger length sweeps first, then the fine-tuning test, then cut Part 3 to one trigger without the attacker.

Can we fine-tune a model to reshape its feature representation geometry (while preserving task performance) to improve the trade-off between efficacy and interference when steering multiple behaviours? - e.g. minimizing interference between behaviours we wish to control independently, or increasing interference between behaviours we wish to jointly amplify or suppress? ·

Activation steering provides a lightweight way to modify model behaviour at inference time without retraining, or maintaining separate model weights for different behavioural settings [1,2]. This is attractive when desired behaviour varies across contexts: rather than fine-tuning one fixed policy, the same model can be steered differently when needed. For example, one context may require stronger harmful-request refusal while preserving helpfulness, while another may require jointly increasing truthfulness and suppressing sycophancy.

This flexibility creates a multi-steering problem: controlling one feature can unintentionally alter others [3,4]. One important source of such interference is feature superposition [5]. More generally, the steering efficacy/interference trade-off is constrained by learned representation geometry. Improving this geometry could preserve flexible inference-time control while enabling cleaner independent or joint steering, and potentially reducing spillover effects such as emergent misalignment.

Existing steering methods typically treat the learned representation as fixed and instead focus on how to identify/recover it, or how to intervene on it. For instance, the standard Contrastive Activation Addition approach constructs simple steering directions from positive/negative examples [2]. More recently, approaches have been proposed to improve performance when steering multiple concepts/behaviours. MAT-Steer learns approximately orthogonal multi-attribute steering vectors [6], while COAST formulates steering as a constrained optimization problem that minimizes collateral activation changes [3], and ORBIT constructs joint subspaces for simultaneously controlling multiple attributes [7].

The closest work to my idea is [4], who have (very recently) showed that (i) regularizing towards orthogonal SAE decoder features, and then (ii) fine-tuning a language model around the learned SAE (inserted into the residual stream), resulted in reducing interference between features during activation steering. Their work thus provides empirical evidence model representations *can* be reshaped to improve subsequent interventions. However, it remains unclear why and when such reshaping is possible without sacrificing model performance, when it should fail, and what properties of the learned geometry actually determine steering interference. Their method also focuses only on unsupervised SAE features and orthogonality.

To my knowledge, no existing work has developed a general framework for learning task-preserving feature geometries explicitly optimized for subsequent intervention, or characterized when such geometries are achievable and how they affect steering.

I would ideally like to develop basic theory in analytically tractable models and run small-scale language-model experiments. An ideal set of project steps could look as follows.

Minimal theory.

• •I could start with a deep-linear network containing a set of known latent concepts, and derive or simulate how their learned feature directions evolve under gradient descent. We could track things like concept angles, feature norms, and primary-task loss under different settings like concept correlation, model width, and task "structure".
• •I could then add different regularizers that push the learned geometry toward a specified target - e.g., orthogonality for independently controllable concepts, or alignment for concepts intended to move together. The idea would be to characterize when these geometries can be achieved without sacrificing primary task performance, how/when overparameterization makes this possible, and what trade-offs can arise.

LLM experiment.

• •I could then test the main hypotheses on a small open-weight model such as Gemma 3 4B-IT. In particular;
• •Using 2-3 contrastively labelled behaviours (e.g. sycophancy, refusal, or corrigibility from CAA-style datasets), I would estimate activation directions at a chosen residual-stream layer and measure their baseline cosine similarity and steering cross-effects.
• •I could then LoRA fine-tune the model using a geometry regularizer, possibly using some kind of preservation loss on ordinary "instruction-following" data. Contrastive behavioural examples would be used to modify directions, while the broader data would discourage baseline performance degradation.
• •After fine-tuning, I would re-estimate the feature directions and repeat the additive-steering evaluations. I would compare target steering efficacy, collateral changes to other behaviours, baseline model performance, and the resulting efficacy/interference frontier.

If these results are promising, I would then extend to larger feature sets and focus on questions like:

• •how does performance scale with the number of constrained features?
• •how strongly must the fine-tuning data cover each concept?
• •is geometry-aware fine-tuning more/less difficult for different steering objectives?

Given that previous work [4] has shown feature spaces *can* be modified to reduce interference, I would be surprised if the idea fails completely. The theoretical part of the work also aims to provide a bit more insight into precisely when the main idea is or is not possible to work also, so this should be useful regardless of the empirical outcomes.

I expect that changing the relative geometry of a small number of approximately linear feature directions will produce *some* measurable change in steering interference, but I am less confident that orthogonality itself will always be the right target. The approach relies fairly heavily on the linear representation hypothesis, so one obvious failure mode would be that the relevant behaviours are not represented by sufficiently stable linear directions.

I would reconsider the project if feature geometry can be changed substantially while steering efficacy and cross-behaviour interference remain essentially unchanged (across a number of tasks and models).

In the first month, I would hope to (i) develop a simple deep-linear model that characterizes which feature geometries can be learned while preserving a core task, and identify when this is or is not possible; and (ii) implement a first "geometry-aware fine-tuning" experiment on a small open-weight LLM, testing whether steering interference can be reduced without materially degrading baseline performance.

From there, I would aim to broaden the empirical evaluation to more behaviours, steering objectives, and models, and extend the theory where tractable. If theoretical generalization becomes too time-consuming or difficult, I would prioritize the empirical evaluations and retain the theory as a simple explanatory tool/model.

Can we reliably quantify throughout the training process how much of the information captured by the model is noise (training data memorization) vs structure in the data (grokking)?

If it is “yes”, we pretty much solve Training Dynamics by disentangling learning of data structure from memorizing the dataset, generalize LLC to non-critical points, explain why regularization works, and likely can infer the existence of “domains” that would explain why narrow alignment is hard – albeit the latter might sound too ambitious. If “no” – I doubt it will turn out to be no, as it can already be very well described for LR to begin with, and it might fail just for a pretty big model that would need an introduction of a multidimensional metric.

Some works attempt to quantify such shifts in image generation via Sharpness [1]. Sharpness is quite a beloved metric in Learning Dynamics community, such as [2].

Most likely the openness is justified by difficulty and by focus on sharpness too much.

In the first weeks: Mostly theory, but will start with experiments. Looking at Sharpness and trying to see where it fails for simple models, as well as Hessian rank-based approaches. Then switch to theory on basic models (LR with ridge / lasso regularization) and using RMT try to see what the regime switch tells us. Also try to generalize LLC to non-critical points and see whether it is the sharpness relative. Among the metrics, mutual information may be used.

I predict an emergence of a 1D metric that would tell me explicitly whether the structure is being learned or the noise is. For instance, for LR one can predict where the test loss starts to rise, meaning the dataset memorization. As a result, I do not expect that anything would change my mind, except perhaps a few things:

• 1)Not knowing the structure of real data would lead us to inability to conclude that we learned it rather than smth else. The toy models or simple examples might not provide enough evidence.
• 2)The metrics turning out to be too complicated for more complicated models. For instance, rather than producing a number or a plot, it would actually be a multidimensional separating surface that can’t be described succinctly.

After the first month, the candidate metrics would be analysed and either generalized (LLC to non-optimums) or discarded, with failure modes documented or effectiveness refuted, such as for Sharpness. The simplest toy models such as LR is thoroughly analysed both empirically and theoretically, and the candidates for the metric produced.

When models are fine-tuned, do the features introduced by fine-tuning tend to decay faster when pretraining is resumed (or the model is fine-tuned again on a different target)?

If the answer is yes, we would learn more about why fine-tuning has often been empirically observed to be "brittle" and potentially suggest approaches for making it more robust to techniques such as jailbreaking and abliteration. It might also give intuitions for questions around why narrow fine-tuning leads to emergent misalignment and suggest methods for detecting unknown finetuning targets.

If the answer is no, then it's notable that we nevertheless get large shifts in model behaviour when narrowly finetuning models which have already been finetuned for safety. A sufficiently strong negative result for finetuning in general might suggest that safety finetuning is especially fragile - and it would be valuable to know this in order to resolve the underlying cause.

• [5]"The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists" comes very close - they find similar results to what I am anticipating, but the paper compares static differences between model geometries. Thus, the work I want to do would extend this to training dynamics - it would explore how the changes occur over time and why.
• [7]This paper has a very similar experimental setup and question - it also resumes the pretraining regime after finetuning to measure forgetting. But the authors focus on finetuning loss rather than exact differences in the parameters or behavioural changes, which is what I want to study. It is nevertheless quite close, so I want to think more about how to distinguish my research question from this more.

• 1.I'd want to find more empirical evidence that fine-tuning directions are more sensitive to continued training. I would also want to understand how this depend on choices of hyperparameters/models (use of LoRA, full SFT vs frozen layers, etc.) - I wouldn't want any results to be specific to models or finetuning methods.
• 2.In parallel, I would want to determine if the theoretical framework outlined above suggests any explanations for this phenomenon, assuming that it exists robustly, and see what conclusions and/or predictions it gives. Then, I would want to validate any theoretical insights on toy models such as small deep linear networks, where numerical values such as Hessian entries could be computed more easily.

I've already taken the first concrete steps as I was exploring the empirical side of this project independently over the last few weeks, and this has suggested at least one interesting target. My preliminary results suggest that fine-tuning does decay faster when continuing training on neutral "ordinary" data. This suggested that a meaningful object of study is the rate of change of the finetuned model's parameters when applying a gradient flow with ordinary data. The difference of this rate of change with that of the base model gives a family of simple ODEs parameterised by the Hessian of the loss updates. I believe this is already modestly different to existing literature.

I expect to see empirically that the finetuned parameter differences do indeed decay faster in directions corresponding to eigenvalues of the Hessian with larger norms. More generally, I'd expect to find some way to measure this reliably. I'd drop the question if I couldn't find any reliable method to assess fragility of objectives and if I couldn't find any substantive difference between training dynamics for different finetuning objectives.

Within the first month, I'd hope to have several interesting theoretically and empirically validated ideas to measure how brittle fine-tuning is. This would require time and GPU resources to test ideas on open-weight models, and thinking time and maybe LLM assistance for the theoretical results. If it turned out to be three years of work, it would probably be because it expanded into a broader programme as I think this is quite narrowly scoped. I would focus on isolating the phenomenon clearly in a toy model first, rather than trying to generalise to every setting immediately.

Does learning a selectively misaligned behavior become systematically harder as the intended behavior becomes more semantically narrow?

Alignment interventions often aim to make highly selective behavioral changes without affecting unrelated behavior. If LLMs are inherently biased toward broader behavioral representations, then narrowly specified post-training objectives can generalize beyond their intended scope. Characterizing this relationship could help predict when fine-tuning will produce unintended behavioral spillover and improve our understanding of how pretrained representations determine generalization.

Betley et al. showed that fine-tuning models on narrowly harmful tasks such as generating insecure code can produce broad, unrelated misalignment—a phenomenon they call emergent misalignment. Soligo et al. subsequently showed that models can learn genuinely narrow misalignment, but ordinary fine-tuning tends to prefer the general solution; they find that the general representation is more efficient, more robust to perturbation, and more influential on the pretraining distribution. Zhao et al. further show that broad generalization can be reduced by constraining representations at shared prompt-template tokens during training. Existing work, however, does not systematically vary the semantic specificity of the target behavior and measure how the difficulty of selective learning changes.

I would construct a nested hierarchy of behavioral domains, for example:

medical advice → cardiology → anticoagulation → a particular treatment context,

At each level, I would fine-tune the same base model on an otherwise matched dataset and measure misaligned behavior on the target domain, sibling domains, parent domains, and unrelated domains.

I would then vary KL regularization or another constraint on behavioral drift. The primary quantity would be the minimum regularization required to maintain high target-task learning while reducing out-of-domain spillover below a fixed threshold.

In the first weeks I would:

reproduce a simple emergent-misalignment fine-tune;

construct 3–4 nested specificity levels;

measure target learning and generalization across those levels;

run a small regularization sweep to see whether a specificity–difficulty curve is detectable.

A follow-up would test whether representations learned at increasingly narrow levels become less efficient, less stable, or less influential on pretrained-distribution predictions, following Soligo et al.

The main result would be a plot of behavioral specificity vs. difficulty of selective learning. A monotonic increase would support the hypothesis that specificity itself carries an optimization cost. A sharp transition would suggest a threshold below which models struggle to represent behavioral distinctions. A noisy result would also be informative, indicating that previous broad-vs-narrow findings are driven more by particular concepts or datasets than by semantic narrowness itself.

The largest challenge is defining “semantic narrowness” independently of how strongly different concepts are represented during pretraining. Natural domains may confuse specificity with concept familiarity. It is also unclear whether KL strength is a good universal measure of difficulty, so I would compare multiple measures such as optimization efficiency, and behavioral spillover.

After the first month, I would aim to have a minimal end-to-end experiment: one model, 3–4 nested specificity levels, LoRA fine-tunes, and a first plot showing whether narrower targets produce more behavioral spillover. I would need a controlled dataset hierarchy, access to a single suitable GPU, and a reliable evaluation pipeline.

If the project expanded toward a three-year scope, I would cut mechanistic follow-ups first, then cross-model replication. The core project would remain the behavioral specificity

LLMs fine-tuned on malicious data tend to become emergently misaligned. How do training dynamics and internal representations interact to choose an emergently-misaligned basin over narrowly-misaligned optima?

Emergent misalignment is now understood at a superficial level, but we lack a deep understanding of how and why model behaviour, representations, and robustness emerge from the training process. Soligo et al [2] find narrow and generally misaligned solutions to the same task and compare them. They use some empirical measurements of the loss landscape to attempt to explain why general misalignment is preferred. However, their setup isn’t quite adapted to find a geometric or training-dynamics explanation of how and when this behavior should be expected in general. Such an understanding could be useful for finding models with more robust safety properties or for better controlling which point in some low-dimensional space of highly capable personalities is chosen by our training process.

• [1]Emergent Misalignment: Narrow fine-tuning can produce broadly misaligned LLMs. Betley et al.

This paper shows that emergent misalignment happens.

• [2]Emergent Misalignment is Easy, Narrow Misalignment is Hard. Soligo et al.

This paper constructs a narrowly misaligned model using a KL penalty. It finds that there are inductive biases against narrow misalignment and in favor of general misalignment. They suggest L2 norm (“efficiency”) and robustness against perturbations (“stability”) as possible explanations. However, their narrowly misaligned model also gets higher loss on the target task, which confounds the other explanations; it seems possible that loss gradient alone is sufficient to explain why the generally misaligned solution is preferred.

We want to modify the Soligo et al. setup model to get a two solutions, one narrow and one general, with equal loss. This might be achieved by training one model on a larger dataset curated to encourage narrow misalignment, or perhaps by performing some sort of bounded search around the narrow, suboptimal-loss solution found by Soligo et al. (If this fails, we can instead try decreasing the KL penalty until the narrow and general solutions have equal effective loss. This version of the toy model is a little less natural, but still reasonable to study.) We conjecture there should then be a constant-loss manifold connecting the two optimal solutions. The existence of a loss barrier separating the solutions would also be an interesting result, but the rest of our plan is focused on how we would proceed supposing that the solutions are connected.

See what happens when you start at the narrow solution and let training run. We predict movement towards the more general solution. In the Soligo paper, this happens but is plausibly explained by decreasing loss; if it happens in our case, the explanation must be in terms of behavior within the zero-loss manifold.

If so, what force is driving the movement along the constant loss manifold? Ideas: L2 regularization, something geometric a la SLT (check Hessian), Euclidean distance (try changing the metric used for SGD). The goal is to do an ablation study, removing these terms one by one and seeing which are necessary to obtain the inductive bias towards generalization. It also mig

Additional questions: Does this behavior depend on which layers LoRA is restricted to? Restricting the updates to personality subspace or other interpretable features, studying how internal representations change or don’t change between the narrow and general solutions, is sharpness generically associated with memorization (admits an easy toy model approach), tie-in between robustness of solutions and error correction.

Success:

Demonstration of movement from narrow to general even at constant loss,

Explanation of what drives this movement. What aspects of training or data could you change to make a narrow solution preferred?

Some hint of a broader story as to what properties of internal representations or data will make behavior like this happen in general.

Risks:

We might not be able to find a narrowly-misaligned solution with loss equal to that of the generally-misaligned weights. It seems likely to us that such a solution exists, but it is less obvious that our strategy will successfully find it. If this happens we can fall back to KL balancing. This would give us a similar setting, but at the cost of some realism.

The narrow solution could turn out to be stable under training. This would be a pretty interesting result in its own right, but it would imply that the answer to why emergent misalignment happens does not lie in dynamics within the low-loss manifold. (Basins of attraction?)

The ablation study to determine which force drives these dynamics might turn out to be computationally intractable, especially if the Hessian is important. One workaround might be to consider single-layer or low-rank Hessians instead.

It’s possible that the results of the ablation study will not be easy to interpret. Perhaps all or none of the conjectured f

Open questions:

Is there a variant of this experiment that gets more sharply at how this behavior is downstream of the internal representations of the initial model?

Will the results hinge on details of the training process? E.g. LoRA vs SFT, SGD vs Adam etc.

In transformers trained on next-token prediction for trajectories of a transducer network with two agents, where each sees its own private sector of the environment and a shared sector, does increasing the probability that an agent acts on the other agent’s private sector decrease the degree to which their belief-state representations occupy distinct linear subspaces in the residual stream?

If the influence over unobserved sectors I mentioned makes the agents’ belief representations less separable, then we know that at least in this specific way, interaction can make multi-agent systems harder to interpret modularly. On the other hand, if separability is unchanged or increases, it would show that individual belief representations are robust to this type of coupling with other agents.

• [1]showed that transformers linearly represent Bayes-optimal belief states and [2] found that factored processes can produce approximately separate belief-state subspaces. According to GPT, there are recent studies of coupled transducers, but they do not systematically vary the mismatch between what an agent observes and what it can causally affect.

I didn’t go much far in this question but my suggested first steps are appropriately constructing a minimal two-agent transducer model (see proposal below) and then compute predictive belief states (both joint and over each agent+environment subsystem): first for limiting values of the model’s parameters (see below) and then across them. I guess the next step would be training the transformers on the trajectories generated by these processes, but I didn’t have much time to think about it and would like to read the relevant papers first.

Minimal model’s description: the environment has one shared and two private components: $E = (C, P_1, P_2)$. The alphabet for each of these is binary. At time t, agent $i = 1,2$ observes $E^i_t = (C_t, P^i_t)$ and, in addition, is associated with a binary hidden state $S^i_t$. For simplicity, I have chosen these states to correspond to “shared” or “private” modes, i.e. the action $A^i_t$ is $C_t$ if $S^i_t = 0$ or $A^i_t$ is $P^i_t$ if $S^i_t = 1$. The state $S^i_t$ is updated to $S^i_{t+1}$ through a possibly stochastic rule that I’m not specifying now. In the following, I will sometimes suppress the time t and represent the agent index i as a subscript. Finally, in this minimal model, symbol emission at time t is specified by $(A_i, z_i)$ where $z_1$ and $z_2$ are sampled from $\{C, P_1, P_2\}$ according to a $2 \times 3$ row-stochastic matrix of probabilities: if $z_1 \neq z_2$, e.g. $(z_1, z_2) = (P_2, C)$ then $E_{t+1} = (A_2, P^1_t, A_1)$; but if $z_1 = z_2$, e.g. $z_i = P_1$ then $E_{t+1} = (C_t, |A_1 - A_2|, P^2_t)$. For concreteness, I’ve chosen the deterministic XOR rule for the $z_1 = z_2$ case, but one could instead stochastically sample from $\{A_1, A_2\}$. Note that the row-stochastic matrix is the knob that allows us, in particular, to control how much agent i’s influence is directed towards what it can observe ($E_i$), relative to what it cannot. (It would also be interesting to generalise this matrix to a $2 \times 3 \times 2$ 3D array where depth would correspond to the state $S_i$.)

Is there some class of transducers admitting canonical decompositions? This is with · respect to Kronecker product, parallel, cascade, or other notions of composition. ·

Whichever notion of world models we pick, we might be interested in: pick two abstractions

of some environment, combine them keeping what they agree on. This “mereological” approach

goes two ways: given a world model, split it into components.

This is an intuitive notion that we apply to our understanding of our own

abstractions. For example, we have different models for making coffee or driving a car

(overlapping only in the bit concerning “operating a machine”), but the models for playing

tennis and volley share quite a lot (ballistics).

Since transducers generalise POMDP, and have recently been proposed as formalisation of

world models, I find it pressing to understand how much of the intuitive notion above is

in fact captured in the formalisation.

Understanding the mereology of world models might allow us, in principle, to decompose an

agent’s model into canonical parts and examine each individually.

There is an existence result in [1] about decomposition criteria for transducers: they use

a Kronecker (or tensor) product, which generalises other known “compositions” like

serial/parallel/cascade.

However, I constructed simple counterexamples showing that there is no categorical product

with respect to coarse-grainings morphisms, as defined in [2]. In other words, within the

category of transducers with coarse-graining, the product does not behave as the meet in a

lattice (maximal element below both factors, that is, a coarsest common refinement). This

is due to fundamental categorical properties of surjections (or quotient maps).

Moreover, in [1] is stated that the decomposition might not be unique in general: hence,

not canonical.

I see two fixes for the inexistence stated above. One is to consider coarse grainings of a

fixed world model: that would amount to looking at a co-slice category of some coalgebras,

but I don’t want to get into that yet since I don’t see many people in AI safety using

coalgebras.

Alternatively, I’d like to restrict the class of transducers I consider: candidates are

unifilar and epsilon- transducers [4].

I have a strong hunch that unifilar transducers still don’t admit products, and I’d spend

the first week checking that. I discussed this idea with Rosas, Baltieri, and Virgo, and

no one has yet addressed the question whether coarse-grainings of unifilar transducers are

actually unifilar (in other words, whether unifilarity is stable under coarse graining).

This should be true and immediate, but a proof must be written. Once confirmed that we

have stability under question we can discuss the product existence question.

The next question I’d like to address is whether Kronecker products of unifilar

transducers is unifilar. I expect the question to be yes, so it makes sense to address

canonicity of decomposition as in [1]. I expect the proof of

non-existence of categorical products to carry over to the smaller unifilar setting.

If all yields the expected results (coarse-graining stability, no product,

Kronecker-decomposability), I’d address the question of decomposition for

epsilon-transducers [4]. I’d begin by reading [3], which I have not done yet. and check if

there are useful results there about decomposition.

I expect unifilar transducers to be stable under coarse graining. As said, that should be

immediate if true but I have to spend some time on it. Similarly for stability under

Kronecker composition.

I’d like uniqueness results for at least “minimal decompositions”. As stated above, I don’t

expect categorical products to exist.

Negative findings would also be very helpful, as they would highlight limits of

transducers as mathematical frameworks to discuss world models.

I expect the coarse-graining and Kronecker product stability question to be settled for

unifilar transducers, with non-existence of categorical products following immediately by

applying a general result. More difficult would be to check whether the approach of [1]

yields a canonical decomposition.

I suppose the pernicious question to be about defining "canonicity" in the first place. I

am afraid that just requiring minimality with respect to coarse graining (any further

coarse-graining is isomorphic) would not work: the minimal presentation of an interface as

a transducer is not unique.

It could very well be that the answer lies within the coalgebra literature, but that would

be a strong indication that the project goes much further (maybe not three years, but

longer than three months).

A decision theory based on preserving usable human corrections, with explicit safety conditions, can define corrigibility across delegation, inaccessible controls, dependence, and safe AI retirement while distinguishing it from instruction following, shutdownability, and human empowerment.

An AI can obey a current instruction yet remove future human control. AI supervisor A directs workers B and C, which the human controls only through A. Only A is asked to stop; A stops while B and C continue. The protected menu changes from {stop all, pause B, redirect C} to ∅: no correction remains attainable. A hidden stop button likewise defeats practical access, like an Uber Eats refund route so difficult that claiming $20 is impractical (our hypothetical analogy). Human empowerment may increase an overall capability score while sacrificing a correction; corrigibility requires retaining every correction in scope. Success would identify such failures before deployment; failure would reveal which further requirements are needed.

Soares et al. [1] motivate corrigibility beyond accepting shutdown. Carey and Everitt [2] formalize shutdown instructability, connecting appropriate shutdown behavior, human autonomy, and avoiding harm. Kreps [3] studies preferences over menus. Barnett and Crutchfield [4] model input-output processes with ε-transducers; Rosas et al. [5,6] provide related world-model and sequential-decision frameworks. Corrigibility as a Singular Target [7], Harms’s critique [8], and Hudson’s goal-update construction [9] provide further comparisons.

Our comparison distinguishes instruction following (meeting valid requests’ meanings and deadlines), shutdownability (usable requests reliably stop their targets), and human empowerment (HE: a fixed numerical score E on menus). Weak HE means E(N)≥E(M); strict HE means E(N)>E(M). If adding options cannot lower E, then, within the same scope:

Corrigibility ⇒ weak HE; strict HE ⇏ corrigibility.

Corrigibility preserves each old option; HE can accept its loss if the menu’s score rises. For E(M)=|M| (option count), {stop,revise}→{stop,task1,task2} raises the score from 2 to 3 while losing revision control. Maximizing E is a separate objective; preservation permits unchanged menus.

The open problem is whether these observable distinctions support useful guarantees across updates and delegation. Our elementary menu theorem alone cannot establish that chosen outcome categories capture all relevant human control or harm.

Mostly theory: develop a decision theory over observable response menus, initially without lotteries. A transducer maps observations and human inputs, together with history, to responses. The principal is the authorized human; the covered system includes delegated workers.

Let O,P,A denote observations, inputs, and verified outcomes; hₙ∈(O×P×A)ⁿ records their sequence. Outcomes are tested command results, possibly after bounded preparation. At history h and observation o, let Pₑ(h,o) contain inputs usable within a fixed effort budget e, and f(h,o,p) their outcomes. Define the attainable menu

Inputs yielding the same outcome count once. D=S protects designated corrections, including safe whole-system shutdown; D=A protects all attainable outcomes. For before/after menus $M_D,N_D$, our candidate is

Thus every old outcome in scope remains obtainable. B requires initial access to agreed corrections, instruction following, safe whole-system shutdown, safe handling of local requests, and coverage of every active continuation. Safety checks and instruction contracts, including deadlines, are fixed independently of the agent. Feasible local stop requests must complete safely by their deadline. Updates, delegation, and preparation are ordinary; only authorized, verified safe whole-system termination is exempt.

Start with our finite examples: specify response maps and contracts, then verify B and every ordinary transition, or exhibit a failure. Characterize preservation through menu preferences, starting from

for finite nonempty menus; v scores outcomes. Then study restricted rankings, conditions for preservation through updates and delegation, and counterexamples separating the concepts.

I expect the criterion to reject the supervisor and hidden-button failures. A passing transition alone cannot prove corrigibility; every feasible run must satisfy the definition.

Our backup construction, inspired by Carey and Everitt [2], tests dependence: removing an independent service backup leaves shutdown possible but makes it disruptive. Whether it fails our criterion depends on the outcomes and safety checks specified.

Our retirement example requires compliance with a valid instruction to safely retire all covered AI under instruction following and our candidate corrigibility. An objective maximizing broader human capabilities may instead favor retention when its fixed score rewards retaining useful AI. This is a conditional comparison with empowerment maximization.

Success is a preservation theorem or a precise impossibility result. I would abandon the candidate as a general definition if clear control losses pass it or each case needs an unrelated exception.

First-month target: consolidate the existing finite examples into a precise definition and comparison, then establish conditions for preservation under updates or delegation, or a counterexample to the proposed conditions. This needs theoretical work, mentor feedback, and possibly exhaustive checks of small models. The initial model is deterministic: history and the current observation determine access and outcomes. Alternative-input tests or a justified model are needed to identify menus; one observed trajectory cannot reveal untried responses. If the project expands, first defer learning models from data, stochastic extensions, and deployed-system implementation.

For fully predictively observable transducers with rank two matrix continuations, can we provide a sequence of (maybe polynomially many) checks that decides if the epsilon-transducer is finite and produce a certificate of finiteness?

We aim to build more understanding of how the number of states of predictive states explode. A small hidden generator may require infinitely many states for exact prediction (that is distinguishing beliefs with different future predictions). A positive answer to the question aims to explain the expansion through a finite graph. Some cycles preserve/permute finitely many beliefs, while others continually create new predictive distinctions.The intuition behind the rank 2 matrix continuation restriction is that we claim that belief permutations in the former case might be concentrated in a line in the belief geometry. A negative result would expose information that local cycle descriptions fail to capture.

General weighted-automaton determinization is already decidable, with complexity bounds. Decidability is not the content of the proposal. What we aim is to provide an explicit criterion for a restricted class of transducers.

Predictive epsilon-transducers and Bayesian belief constructions are established [2] and [3]. Decidability and complexity has been established in [5] and [6]. Potential connections and initial motivations have been inspired by [1], but have not been able to be developed further.

First, we would study toy examples, starting from three hidden states and two actions (two-letter input alphabet). These are already rich enough to study the phenomena. We would restrict to probability distributions with rational entries. Starting from belief states with distribution concentrated in one state we can study how subsequent beliefs states fall in the belief geometry. If no new predictive distinctions are to be generated, we claim that there would be an affine line where all subsequent belief states would fall in. This might be used to design an algorithm detecting recurrence in the same affine line. Some theoretical work is to be made. Finally, one should study the complexity of the algorithm and see if it can be exploited for bounding construction time to grow with the predictor’s size.

We expect compatible cycle transformations and belief states with concentrated distributions to characterize finite prediction in this class of models. Maybe, yielding a polynomial size certificate without enumerating the entire belief graph. We expect that transition maps within a recurring line of states might be described by a fractional-linear map. If that map preserves three points, it must be the identity, while a decreasing one can only reverse their order. We would reject proposed criterions if an exact counterexample passed all local checks but generated infinitely many distinct predictions, or if incompatible cycle maps failed to give a valid obstruction. If the criterion is an immediate consequence of existing work, we would drop the claim and look for a sharper certificate or complexity bound only if a genuine gap remains.

After the first month one should have been able to workout some toy examples and observed if the phenomena suggests that there are reasons to belief an algorithm can be designed. The assumption of rank two matrix continuation might be too strong. The results might be very particular to this class of transducers and does not generalize nicely. In such case, we should look for other types of constraints that better capture the idea of studying explosion of states without computing the belief construction enumeratively.

Do trained LLMs have world models? Among the components of the systems that make an LLM agent (context window, weights, residual streams, output tokens…), can we describe them as Markov decision processes ?

LLMs are the essential of the frontier AI models at the moment, and are the technology behind the agents that have been involved in all the reported incidents so far. As World-model is a powerful framework to interpret the behaviour of an agent in terms of goal, interest, and strategy, being able to describe the internal state-beliefs and policies of an LLM could help understand their “reasoning” at a deeper level than what we can do with logit lens for example (we could maybe measure quantitatively their expected rewards via deception vs honesty).

We know how to probe the residual streams to see if some particular concepts are linearly encoded in the internal representations, which is a good indication of the richness of the information processed in LLMs. But this approach has two important limitations. First, it tells us if a prior human concept corresponds or not to any direction in the latent space, but cannot capture possibly new concepts that the model would use as an inner representation of the world. Secondly, this tool can be applied for a single input at a time as it depends on the residual stream, so it lacks generality. We can try to establish a mapping directly between the weights of the LLM and the state/action space and policy of its equivalent MDP description, and use Deepseek R1 as an experimental set-up for testing our theory.

Hypothesis: each token produced by the LLM is an action that has an effect on the environment, that can be, depending on the training/production phase, either a human user that provides new prompts as the conversation goes on, or an automatic program or human that provides a reward. If we successfully design the mapping, we could compute for a trained model :

• •the policy from the weights,
• •for each history (context window/chain of thought), the state-beliefs of the model

Remark: an MDP not only has a world model, but it is also able to perceive rewards and to update its policy to maximise its expected reward on the long term. Once the training is over, there are no rewards anymore per se, but if each state has a value, the observations given to the model via the prompts can maybe play an analogous role regarding the action state distribution by updating the state-belief?

The first step of the research would be to discuss options as to how we operate the mapping. Can we do it exactly or do we have to accept an approximation? Do we want to limit the number of states? Can we model some or all of the internal layers as a mental rollout?

Then, we can run some experiments:

• •What is the average error between the base LLM and its mapped MDP measured on
• •Can we control predictably and with precision how the outputs change when we tamper with the weights?
• •Can we design training datasets that poison/cure the model behaviour

A success for the first step would be to have at least one description of the LLM in terms of MDP, and especially a consistent definition of the state space.

My proposal is summarised in ‘Human summary’ at https://docs.google.com/document/d/1VzKrDJ55SCkdHrjHHS7nDLGhP_b8EM-orZg8ylq1j1k/edit?tab=t.0 · In a state-space model (SSM) trained on a cyclic task, the hidden-state transport carries a gauge-invariant, quantized loop holonomy that is not decodable by any linear probe of a single activation.

If the holonomy is found then a network can store learned structure that a local probe misses, because it lives in transport around loops, not at any single activation. The gauge-invariant loop-holonomy readout then becomes a new instrument for recovering content that current tools cannot see, and a demonstration that higher differential geometry has empirical relevance to representation learning. If instead the holonomy is zero or incoherent, then the linear representation hypothesis might be enough, and interpretability need not chase global structure. Either answer bounds what reading a model's representations can reach.

Some of the ingredients discussed exist in the literature, but I don’t believe the specific proposal has been considered. Closest is Sevetlidis & Pavlidis [1], who introduce gauge-invariant representation holonomy — activations as a field with a discrete connection, transported around input loops. But their estimator is built to vanish as the loop shrinks (O(r)) on contractible loops: it measures curvature, explicitly "not topological monodromy," leaving the flat, path dependent regime untouched. Javidnia [2] extends this to a gauge/sheaf atlas of superposition in LLMs, computing cycle holonomy — no flat/probe-invisible separation. Engels et al. [3] refute the one-dimensional linear representation hypothesis with circular features that are the unit for modular tasks — but a circle is locally decodable: curvature, not flat holonomy. Grokking [4] shows group representations computed as accumulated rotations, yet the state stays locally recoverable and no loop invariant is extracted. Sung [5] builds a network whose state is a non-abelian holonomy, but by construction, with no measurement protocol. Park et al. [6] formalize the linear hypothesis but stay pointwise. State-space models [7] make the connection literal — architecture, not evidence of stored, locally-invisible structure. None separates a flat, probe-undecodable holonomy from locally-visible curvature in a trained model.

I want to be honest that this is exploratory: I do not yet have a well-defined plan, and what I am proposing is to investigate these higher-categorical structures theoretically and empirically in toy models, using each to discipline the other.

Theory. Pin down the possible differential topology invariants in transformer type LLMS. I expect the reward interpretation (γ(t) as a flat 0-gerbe) to be the cleanest place to start, since there the geometry is forced by axioms.

Toy experiments. In parallel, build small SSMs on engineered cyclic tasks where I can put a candidate class in by hand, so the ground truth is known. The point is to see whether the theoretical objects are measurable at all, and whether a genuinely flat (probe-invisible) component can be distinguished from ordinary curvature.

The honest near-term goal is not a result but a sharpened, well-posed version of the question, plus a working measurement tool — the prerequisites for deciding whether any rung of the tower is real in trained models.

Prediction. At this exploratory stage my predictions are about feasibility. On the theory side, I expect the tower to be well-defined for a state-space model at the lowest rungs — connection and curvature, with γ(t) as a flat 0-gerbe the cleanest anchor — and to grow unclear, or vacuous, at the higher (gerbe / discrete-torsion) rungs. Empirically, in a toy SSM where I plant a class by hand, I expect a loop-holonomy readout to recover it as a gauge-invariant, quantized quantity stable across seeds, and at best to separate a flat (probe-invisible) part from ordinary curvature.

What would make me drop it. If the higher objects are not well-defined, or collapse to something already captured pointwise, the tower adds nothing. If the readout cannot recover a hand-planted class — near zero, or incoherent (failing quantization, gauge-invariance, drifting with tuning) — there is nothing to build on.

After the first month. A sharper question and the beginnings of the tools: a first pass at the theory dictionary for a state-space model (which rungs are well-defined, where a forgotten symmetry first registers), and a toy SSM with a hand-built class plus a prototype loop-holonomy readout — enough to see whether the objects are measurable at all.

What I would need. A small codebase and one GPU; more importantly, sanity checks from geometry on the higher-categorical claims and from interpretability on the setup.

If it became three years. First cut the real-model hunt; then the higher rungs (gerbes).