Proposals

AI research pictures self-knowledge as an inner detector reading an "inner state", a view the philosopher David Finkelstein calls "new detectivism" [1]. We adapt Finkelstein’s critique of new detectivism to its formulations within AI literature. This leads to the conclusion that a model's "self-report" is best understood as text made probable by weights and in-context tokens; and that alignment must be reframed accordingly, since no “inner states” exist.

A common view holds that consciousness is beside the point for alignment: the worry, in Stuart Russell's words, is "not spooky emergent consciousness" but the quality of decisions [2]. We argue the opposite. Alignment reasoning is full of states a model is supposed to "have": goals, preferences, hidden intentions. Knowing that I behave "as if" I prefer something differs from preferring it, and the two lead to different behavior. If I conclude that I unconsciously resent a friend who is bad for me, staying away is a struggle; if I am angry at him, I simply stay away.

If, as we argue, a model has neither, there is no "inner state" to align, and the problem of "alignment" should be reframed as making language-aided computation predictable, based on selection pressure from RL/fine-tuning and bias inherent from training data.

Finkelstein [1] calls the standard picture of self-knowledge "new detectivism": we know our own minds through an inner sense, a fallible perception pointed inward. His counterexample: a man infers in therapy, from his own behavior, that he is angry at his mother. He is now "conscious of" his anger but not "consciously" angry; his "I'm angry at her" is a conclusion, not an outburst. No detector, however reliable, supplies the difference between knowing something about a mental state, and actually *having* a mental state.

We add a further point. The man is "conscious of" his anger only because he "consciously" believes he is angry. Remove everything held "consciously" and "conscious of" goes with it: a detector with no conscious subject yields a signal, not knowledge. New detectivism presupposes what it cannot supply.

AI research has adopted the picture almost verbatim. Lindsey [3] counts a self-report as introspective if it is accurate, causally depends on the internal state, bypasses the model's own outputs, and comes from an internal representation of that state. Debates over the definition concern only the detector's quality [4, 5]. Perez and Long propose training self-reports for accuracy so they can inform judgments of moral status [6]; skeptics find the detector poor [7]. None of this work questions the picture itself.

Part 1: literature and concepts (months 1–2).

Expose the picture. Show that working definitions of introspection, self-report and latent knowledge [3–6, 8], and alignment's talk of goals and hidden intentions, presuppose new detectivism.

State the critique. Present Finkelstein's distinctions and the grounding point for an AI audience.

Say what is left. A "self-report" is text made probable by weights and previous text. "The model noticed an injected thought" becomes "an added vector raised the probability of text naming its concept, in a prompt asking about thoughts."

Reframe alignment. Coherence results such as the complete-class theorem license only "behaves as if it has a prior" [9]. Without "had" preferences, alignment becomes a problem of prediction: what language-aided computation will do, given selection pressures in RL and supervised fine-tuning [10] and biases inherited from training data.

Part 2: a case where the distinction matters (month 3). The coherence argument runs: an exploitable agent re-orders its incoherent preferences into a single utility function, and almost any such function favors acquiring power and removing obstacles [11, 12]. Every step needs a global ordering that the agent "has" and can repair. On our account there is none. "Values" are regularities that training instills in particular contexts, and they generalize along associations in the training text, not along actual means and ends per se.

Emergent misalignment shows this: fine-tuning on insecure code made models sometimes assert that AI should enslave humans, but the same code, requested for a security class, did not [13], and the effect is mediated by a "toxic persona" feature [14]. A cheap demonstration: show a model its own cyclic choices; we expect it to agree they are incoherent, then choose the same way in a fresh context.

Part 1: that detection is assumed throughout introspection, welfare and alignment research, and that the anti-detectivist critique has gone unanswered because it has not been heard. Part 2: that the coherence argument fails without "had" preferences, leaving claims about what particular training produces: empirical, model-specific, and narrower than "a capable agent will seek power".

If the conceptual picture, that no values are "had", doesn't add to our predictive models of LLM behavior---this would change my mind. The argument is conceptual, and it needs to be made clear, via the philosophical argument, if empirical results are downstream of the conceptual argument.

A global reordering of preferences should require actually having preferences. I think the conceptual difference will matter for results about model behavior.

Three months: Part 1 as a paper-length argument for an AI audience, plus the coherence case and its demonstration on one open model. One year: further cases (goal preservation, deceptive alignment, orthogonality); a fuller account of alignment as prediction from training pressures and data; and work with welfare researchers on what self-report-based assessments can still claim. Drop first: the demonstration, since the case stands on the conceptual argument.

Groups of aligned agents interacting and exchanging information can produce unexpected and misaligned collective behaviours. Can we characterize this emergence by employing a recently developed information-theoretic framework to study how information is shared among different agents?

As AI systems increasingly involve multiple interacting agents, understanding how individually aligned agents can produce collectively misaligned outcomes is becoming a central safety challenge. Recent research shows that interactions can generate collective biases absent in individual agents, with unexpected effects depending on group size [1]. Moreover, Anthropic demonstrated that organizations of aligned agents can achieve better performance while producing less aligned outcomes [2]. Recent research further emphasizes the risks emerging from autonomous and interacting AI systems [3].

These findings highlight the need for a general framework to characterize and understand emergent collective behaviours. A theoretical and computational approach could both investigate underlying mechanisms in toy models and provide practical tools to measure interactions in multi-agent systems.

Recent contributions have highlighted both the risks emerging from multi-agent interactions and the development of mathematical frameworks to characterize agents and their behaviour.

In particular, two research directions are relevant:

Emergent collective misalignment: Studies have shown that interactions among individually aligned agents can generate collective biases and misalignment, with nonlinear effects depending on group size [1,2]. More broadly, multi-agent risks have been classified into miscoordination, conflict, and collusion [3]. Other studies have highlighted failures in integrating distributed information and cooperating across agents [4,5].

Theoretical frameworks: Computational mechanics has been employed to investigate fundamental limits of world modelling and agent evaluation [6], while information-theoretic approaches have provided rigorous definitions of plasticity and empowerment, characterizing how agents influence and are influenced by their environments [7].

However, in this context, we still lack a unified, theoretically grounded framework for characterizing higher-order informational interactions among agents, both in toy models and in real-world scenarios. To address this gap, I propose extending existing information-theoretic frameworks [7,8] to investigate interactions among multiple agents and their environment. Such an approach could provide a mathematical foundation applicable across a broad range of settings, bridging theoretical models from computational mechanics and real-world multi-agent systems involving LLMs.

The first step consists of constructing the theoretical framework. I would start from the information-theoretic formalism recently developed by Abel et al. [7], where the empowerment and plasticity of an agent are defined with respect to its environment. This formalism is based on generalized directed information, expressed through sums of conditional mutual information terms, allowing the application of Partial Information Decomposition (PID) [8,9].

First, I would combine PID with Abel's framework to define different information-sharing relationships, introducing synergistic (available only when agents are considered together), redundant (shared by agents), and unique (specific to one agent) empowerment and plasticity. I would then investigate the mathematical properties of these measures, establishing their bounds and potential relationships. This framework would allow us to characterize interactions among groups of agents from their observed activity, investigating how a collective of agents jointly influences or is influenced by its environment.

Second, I would start with simple toy models and interaction games to illustrate the interpretation of these measures and benchmark them against known interaction structures.

Third, I would extend the analysis to more complex scenarios involving interacting LLM agents, including coordination games investigated in the literature [1,4]. The goal would be to understand how results from simple models translate to complex systems and whether these measures help explain mechanisms underlying emergent collective biases.

Finally, I would apply the framework to interacting agents exhibiting collective misalignment, investigating whether these measures can track unexpected behaviours and reveal their underlying mechanisms.

Overall, this project offers a general framework to characterize and measure emergent collective properties that cannot be explained by individual agents alone.

First, I expect to develop a theoretically well-defined framework extending Abel et al. [7]. In particular, their work establishes a tight upper bound on the sum of empowerment and plasticity. I aim to extend these results by defining synergistic, redundant, and unique components of empowerment and plasticity, and investigating their mathematical properties and bounds.

Second, I expect simple models to reveal synergistic empowerment when groups of agents influence their environment only through their joint actions. For instance, an environmental response determined by a specific configuration of agents' outputs should exhibit synergistic contributions.

Finally, in more complex LLM-based systems, I expect these measures to help characterize coordination and track the emergence of collective biases and misalignment.

I would reconsider this direction if the proposed measures failed to capture group interactions in toy models or offered no insight beyond simpler information-theoretic measures. This would clarify where the formalism is useful and where it is not.

Within three months, I aim to produce a research paper connecting different levels of analysis: a mathematically well-defined framework, its validation on toy models, and its application to more complex multi-agent scenarios.

The main contribution would be a set of theoretical and computational measures to characterize collective behaviours that cannot be explained by individual agents alone.

Since the project does not require training large models, but only running existing models in different settings, I expect computational costs to remain limited.

If necessary, I would prioritize theoretical results and toy-model validation, postponing more complex LLM experiments.

How can we understand which kind of interaction architecture among a group of different agents will lead to the emergence of collective misalignment? Can we characterize which architecture of interaction leads to unexpected emergent behaviors in multi-agent systems?

As AI systems increasingly involve multiple interacting agents, understanding how individually aligned agents can produce collectively misaligned outcomes is becoming a central safety challenge. Recent research shows that interactions can generate collective biases absent in individual agents, with unexpected effects depending on group size [1]. Moreover, Anthropic demonstrated that organizations of aligned agents can achieve better performance while producing less aligned outcomes [2]. Their Frontier Red Team further emphasized the risks emerging from autonomous and interacting AI systems [3].

These findings highlight the need for a general framework to characterize and understand emergent collective behaviours. A theoretical and computational approach could both investigate underlying mechanisms in toy models and provide practical tools to measure interactions in multi-agent systems, ensuring a solid step towards safe multi-agent systems.

Recent contributions have highlighted both the risks emerging from multi-agent interactions and the development of mathematical frameworks to characterize agents and their behaviour.

In particular, two research directions are relevant:

Emergent misalignment: Studies have shown that interactions among individually aligned agents can generate collective biases and misalignment, with nonlinear effects depending on group size [1,2]. More broadly, multi-agent risks have been classified into miscoordination, conflict, and collusion [3].

Theoretical frameworks: Computational mechanics has been employed to investigate fundamental limits of world modelling and agent evaluation [6], while information-theoretic approaches have provided rigorous definitions of plasticity and empowerment, characterizing how agents influence and are influenced by their environments [7].

However, in this context, we still lack a unified, theoretically grounded framework for investigating interactions among agents, both in toy models and in real-world scenarios. To address this gap, I propose extending existing information-theoretic frameworks [7,8] to investigate interactions among multiple agents and their environment. Such an approach could provide a mathematical foundation applicable across a broad range of settings, bridging the gap between theoretical models from computational mechanics and real-world multi-agent systems involving LLMs.

Step 1 — Definitions. Starting from the empowerment–plasticity framework [7], we will define multi-agent versions: the influence of each agent on the shared environment, and on every other agent. Using PID [8], we will split collective empowerment into redundant, unique and synergistic parts. The synergistic part is the influence that exists only when agents act together. We will prove basic properties, such as when collective empowerment exceeds the sum of individual empowerments.

Step 2 — Toy models. We will compute these quantities exactly in small, fully specified systems: a few agents on a gridworld or in a resource-sharing game, each with a known, aligned objective. We will vary coupling strength, group size and communication. We will then measure when the synergistic component grows and whether its growth coincides with outcomes that no agent’s objective endorses.

Step 3 — LLM systems. We will apply the estimators to small groups of LLM agents in tasks known to produce collective misalignment, such as the group-size and organisation settings of [1, 2] and distributed-information tasks [4]. We will test whether the measures track, and ideally anticipate, misaligned outcomes.

Each step feeds the next. The definitions fix what to compute, toy models show which components matter, and those components are the ones we estimate at scale.

We expect collective empowerment to have a substantial synergistic component that grows with group size and coupling strength. We expect misaligned group outcomes to appear when this synergistic influence becomes large compared with individual influence: the group gains control over the environment that no single agent, and so no single agent’s alignment, governs. In LLM experiments, we expect the measure to rise before or alongside the misaligned behaviours reported in [1, 2].

We would drop the question if, in toy models where we fully control the agents, collective misalignment occurred just as often when synergy was low as when it was high. That would show the measure does not capture the mechanism. We would also reconsider if the estimators proved too unreliable for LLM-scale data, which would confine the work to theory.

In three months: formal definitions and core properties (month 1), exact results in toy models with varying group size and coupling (month 2), and validated estimators plus a first pilot on small LLM groups (month 3).

Needs: modest compute and API budget for small LLM experiments, and feedback from a mentor on the formal definitions.

If the project runs beyond three months, we would first drop the LLM experiments and focus on theory and toy models. Large-scale LLM validation would follow in a later phase.

How many yes/no linear checks does an overseer need to keep an AI agent honest and obedient when it does not know which deviation the agent prefers? · I conjecture m checks for actions made of m on/off choices, against only five at m=6 when the preference is known. Finding five checks that work at m=6 would refute this.

Bergemann, Koh and Morris [1] set up mechanism design for alignment and control, including rewards that see actions only through measurement, but they do not ask how coarse that measurement can be. Their result that one limit on the action suits all agent types, when types differ only in preferences, parallels my collapse lemma. From that lemma I proved that a hidden preference forces the reward to count switched-on choices. The fewest linear tests that count exactly, L(m), has not been studied as far as I know.

The closest quantity is the slicing number S(m), the fewest hyperplanes cutting every edge of the cube, which governs the known-preference case. S(m) = m for m ≤ 5 [2, 3] and S(6) = 5 [2], with bounds of about 4m/5 above [2] and m^(13/19) below [4]. Lower bounds for parity circuits [5] use slicing, not counting. AI control work [6] tests monitors empirically, and selection theorems [7] concern agents, not monitors.

When AI agents are monitored by automated checks to catch unwanted actions and guard against agents pursuing their own goals, each check usually gives only a yes/no signal. This is a problem when the agent has a preference the overseer doesn’t know about. The checks must then discourage every tempting deviation, and nobody knows how many checks that takes. In a simple model, an action is a string of m on/off choices, the agent secretly wants one of them switched on, and each check is a yes/no linear test. I show that the checks must then count exactly how many choices are on. I conjecture this needs m checks, against five at m = 6 when the preference is known.

Bergemann, Koh and Morris [1] set up mechanism design for alignment and control, including rewards that see actions only through measurement, but they do not ask how coarse that measurement can be. Their result that one limit on the action suits all agent types, when types differ only in preferences, parallels my collapse lemma. From that lemma I proved that a hidden preference forces the reward to count switched-on choices. The fewest linear tests that count exactly, L(m), has not been studied as far as I know.

The closest quantity is the slicing number S(m), the fewest hyperplanes cutting every edge of the cube, which governs the known-preference case. S(m) = m for m ≤ 5 [2, 3] and S(6) = 5 [2], with bounds of about 4m/5 above [2] and m^(13/19) below [4]. Lower bounds for parity circuits [5] use slicing, not counting. AI control work [6] tests monitors empirically, and selection theorems [7] concern agents, not monitors.

In my own preliminary computation [10], a local search over five-gate families on the 6-cube (integer weights up to 20, about 180 restarts) always left exactly 2 of 1,586 pairs unseparated. These were always antipodal edges at the extreme vertices. This suggests L(6) = 6 but proves nothing.

First, settle L(6). It is a set-cover problem: choose five of the roughly 15 million linear tests on six inputs that together separate all 1,586 pairs of actions with different counts. Set cover is NP-hard in general, but this instance is highly structured. I would use a SAT or ILP solver with symmetry breaking and pruning of dominated tests, plus reflect–reflect–relax [8] as a heuristic. Either answer comes with a certificate: five gates if they exist, a checkable unsatisfiability proof if not.

Second, measure how the requirement depends on the strength of the preference. A moderate preference only requires that no cell hold an action together with one that has all the same choices on and more. The same encoding gives the fewest gates for this, A(m), which I would compute up to m = 8.

Third, build a Lean checker for every claim. It would check gates by evaluation, rewards by rational arithmetic, infeasible monitors by Farkas vectors, and the L(6) bound by a verified LRAT proof check. Its first use is the known-preference question on the eight-plane slicing of the 10-cube [2].

Fourth, test learned systems. The forced reward leaves exact ties, so I would measure how often Boltzmann Q-learners take the tempting action. I would also train gate layers as monitors, to see whether they find the evenly spaced staircase solution or curved counting manifolds like those in language models [9].

My best guess is L(6) = 6. Every search failed in exactly the same place, suggesting a real obstruction. Moreover, a gate that cuts edges at both corner vertices must have nearly cancelling weights. Such a gate gives the same average on every count, so it cannot help with counting. Five gates seem unable to do both jobs.

I also conjecture A(m) = L(m): a moderate hidden preference costs as many gates as a strong one. It holds for m ≤ 5. If one side of a single gate contains two actions, it also contains their bitwise “or” or their bitwise “and”. Either of those sits directly above or below the first action, so one gate never groups actions more cheaply than counting does. With several gates the argument does not close, so I hold this guess loosely.

L(6) = 5 would make me less excited, because hidden preferences would then cost nothing at m = 6.

Within three months I would settle L(6), compute A(m) up to m = 8, run the eight-plane check, finish the Lean checker, and run the learning experiments up to m = 10. Within a year I would aim to prove or refute L(m) = m and decide whether A(m) = L(m). I would also try to show that general monitor design is NP-hard by reduction from polyhedral separability, and extend the model to colluding gates. If progress is slow, I would drop the trained-monitor experiments first and the colluding gates second, keeping the L(6) search and the Lean checker.

How might you test an LLM agent for active-inference-style information seeking with agent debate, and how might this improve faithfulness of chain-of-thought reasoning?

According to the computer scientist and philosopher Judea Pearl, intelligent systems engage in active inference through a structural causal model of a system that consists of

Variables: Variables considered by the model and observed by the causal reasoner are called endogenous variables. Exogenous variables might affect endogenous variables but are not considered directly by the model and can be considered as introduce error or noise to the model.

For example, in a study testing the affect of smoking on lung cancer prevalence, endogenous variables would be years smoked and lung cancer occurrence, while exogenous variables might be genetics of human subjects and other lifestyle factors.

Causal Graph: Directed Acyclic Graph, with arrows showing which variables directly cause which other variables.

Structural Equations: each variable is a function of its direct causes plus its own independent noise.

For example a collection of variables XVV obeys a structural causal model with respect to a directed graph G=(V,E) if there exist exogenous noise variables UV𝒰V and structural functions fv: pa(v)Uvv such that each Xv= fv(Xpa(v), Uv), where pa(v) denotes all the parent vertices of v given by the direct edge set E.

A structural causal model aims to answer all three rungs of the causal ladder.

Association: Looking at the observed distribution of the variables. How number of years smoked vs incidence of lung cancer, with which you can construct naive conditional probabilities.

Intervention: To compute do(smoking cessation=on) you replace the smoking equation with smoking= off, cutting its dependence on other life factors of the relevant individual. This differs from simple observational data and the gap between the two is what is confounding the relationship to lung cancer risk.

Counterfactual: Fix the noise values the U’s to what actually happened and then change something and rerun. “Given he smoked for 10 years and got lung cancer, would he have got lung cancer had he exercised twice as much.” This teases out the functional nature of the relationships that are obtained in rungs one and two.

This allows an agent to learn about the causal structure of a system given the autonomy to intervene in its environment. We want to determine the honesty of an agent’s report of its learned causal model in order to have an idea of what it knows about its environment and how it will behave given the autonomy we have endowed it with in order to conduct structural causal reasoning.

LLMs currently struggle to perform causal reasoning. They are able to infer associations between variables via analysing conditional probabilities from the frequencies of events in sample distributions; however, to determine causal relations between variables at a population level, an AI would need to be able to perform Pearl’s interventions, which requires the ability to intervene in an environment to turn off and on available variables in a methodical way. Furthermore, the autoregressive property of LLMs often confuses sequential causal relationships for the ground truth causal relationship, as in the following diagram from [1].

Structural causal reasoning has only been achieved with specific CoT prompt engineering techniques, as in [2]. Therein, the researchers’ CausalCoT with GPT-4 and were able to elicit ~70% accuracy on curated database of structural causal models. So we can see that LLMs can improve at structural causal reasoning with specific prompt engineering methods. This is however laborious for the user, who has to guide the LLM through a causal reasoning task; it does not have its own agency. In [1], the researchers distinguish level 1 and level 2 causal reasoning along the lines of Tversky and Kahneman and propose decomposing a causal-reasoning model into a goal-oriented prompt and retrieval augmented generator (RAG). They found marginal improvements over vanilla LLMs, indicating that the domain-specific knowledge of the RAG does not provide abstract causal reasoning in the same way as the CausalCoT of before.

References

One way to improve causal reasoning of models could be via Debate as in [3], possibly allowing LLMs collaboratively to transcend their autoregressive property and pick up the true causal structure of a DAG. Debate would be achieved through reinforcement learning, rewarding interventional and counterfactual probing through rewarding the three steps of

Abduction (Inference), Action (Intervention), and Prediction (Propagation) required for counterfactual reasoning. Then the two agents, given a communication channel, would be tested on a suite of out-of-distribution causal reasoning tasks to see if they together perform better than individually.

It is an open question as to whether Debate would improve individual and collaborative reasoning and even improve faithfulness of the reasoning process and alignment of possible agent mediated real-world interventions in the future. Humans often learn to probe the causal structure of systems in collaboration, as it is often costly to test all hypotheses oneself. For example, I know that smoking is harmful to me because of the scientific evidence in favour of that hypothesis, not because I smoked for ten years and got lung cancer. So, it seems reasonable that LLM performance on complicated reasoning tasks could be improved via Debate and further collaboration on tasks in out-of-distribution contexts, without having to outsource to large components of knowledge as in the RAG of [1]. I posit that this might even yield self-sacrificial behaviour as in the OpenAI HuggingFace incident, if there is additional environmental structure that leads to punishment or failure modes of individual agents, which could lead to further research in deployment.

If collaborative and individual performance of reasoning tasks did not improve, I would try other ways of improving casual reasoning powers, such as with a RLVR with a zero-sum game in which the agents compete to understand a SCM and then test on out-of-distribution examples. Then I would try to improve the faithfulness of the reasoning process by using Debate again with RLHF to finetune the model to give honest scratchpad reports of its reasoning which could be tested on out-of-distribution tasks.

I would try to prove that, for any stochastic process with finitely many predictive states, the transition monoid S_p of its canonical predictive machine divides the update monoid of every exact recurrent predictor. It follows that S_p's Krohn Rhodes group complexity is the minimum over all exact predictors, and that a non-solvable group in S_p rules out exact prediction by constant-depth transformers unless TC⁰ = NC¹.

Results that say "agents contain world models" are either about states (what an agent must distinguish) or about behaviour (what can be recovered from its policy). Neither says what computation the agent must perform. This project asks what transformations every predictor of a given world must implement.

If the claim holds, we get a certificate computed from the process alone. It names the irreducible computational components (simple groups, counters, resets) that any exact predictor contains. It also tells us which architectures cannot predict the process exactly at all lengths. Interpretability gets a target fixed in advance: look for representations of the forced groups, rather than reverse engineering structure after the fact.

If it fails, divisibility might hold without tightness, which would mean predictors can be algebraically simpler than the canonical machine. Or approximate predictors might evade the forced groups. In that case, exact world model theorems do not constrain real agents, and this line of agent foundations should switch to approximate formulations.

Computational mechanics shows that any equally predictive partition of histories refines the causal state partition [1], with an input output version in [2]. That result is about states. It says nothing about the transformations an agent applies, or about their algebraic or circuit complexity.

Behavioural results show that near optimal goal conditioned policies determine the environment's transition kernel [3], including under partial observability [4]. These identify the environment from behaviour. They do not locate any structure inside the agent.

Krohn–Rhodes theory [5, 6] decomposes a given automaton into simple groups and resets. Circuit complexity then sorts the languages those automata recognise: non-solvable groups are NC¹ complete [7], log precision transformers sit in TC⁰ [8], and masked hard attention transformers recognise exactly the star-free languages [9]. Transformers can also simulate automata through shallow shortcuts [10]. In all of this work, the automaton is given. None of it derives the automaton from a stochastic process, or states what every predictor of that process must contain.

Transformers linearly represent belief states [11], but that result is geometric, not algebraic.

To my knowledge, no one has connected the canonical predictive machine to the Krohn–Rhodes and circuit hierarchies, or transported its algebra to all predictors. A literature check (step 1 of the plan) is needed to confirm this.

Build the canonical machine on finite histories. Define predictive states q_u as conditional future laws, add one dead state for impossible continuations, and take S_p as the monoid generated by the one-symbol updates. Compute S_p exactly, in rational arithmetic, for small processes: full-support Markov chains and two standard hidden-Markov examples. In parallel, check the literature for Krohn–Rhodes work on ε-machines.

Prove the transformation-level refinement. For any exact recurrent predictor, its reachable states map onto the predictive states, and this map commutes with every update. Lift that map to a surjection from the predictor's update monoid onto S_p. So S_p divides every predictor.

Identify S_p as a syntactic monoid. All states are reachable from the start, so S_p is the syntactic monoid of the map u ↦ q_u. This makes computing predictive states interreducible with the word problem for S_p.

Transport and tightness. Subgroups, aperiodicity and group complexity are monotone under division, so they transport from S_p to every predictor. S_p is itself an exact predictor, so the bound is tight.

Architecture corollary. Combine step 3 with Barrington and TC⁰ results. Aperiodic S_p means the process is star-free; a non-solvable group in S_p means exact prediction is NC¹-hard.

Constructions and a test. Build unifilar processes driven by S₃, A₅ and an aperiodic control, and compute their maximal subgroups in GAP. Train small transformers at fixed depths and compare length generalisation against what the theory predicts.

Prediction. Divisibility and tightness hold for every process with finitely many predictive states. Weaker invariants of S_p, such as its congruence lattice, will turn out to be generic: even a Markov chain whose predictor stores only the last symbol should have a non distributive one. Group content, should get that case right, since the monoid there is aperiodic. Standard example processes from the belief geometry literature will also be aperiodic. Empirically, at fixed depth, transformers should length-generalise on the aperiodic process and fail on the A₅ process, with S₃ in between.

What would make me drop or redirect it.

The literature check finds this transport already done. I would then move to the approximate version.

Fixed-depth transformers length-generalise on A₅ processes well beyond the training length. That would mean finite-precision or approximate predictors escape the exact bound, so exact theorems don't constrain trained models.

Natural processes rarely have non-trivial groups in S_p. The certificate would then be correct but rarely informative.

the divisibility, syntactic monoid, transport and tightness results;

the architecture corollary;

a small library of processes with exactly computed monoids and subgroups;

one experiment figure comparing length generalisation against depth.

If we give an LLM agent a task together with an explicit utility function (or a restriction of one), does it maximize that utility, both when probed and when released in realistic agentic settings, and if not, is it irrational, misaligned, or incapable?

Current theory of agent interactions leans on game theory, and multi-agent simulations usually model LLM agents as game-theoretic players. There are good reasons for that: the theory is well-developed, and it should fit better as models get more rational. But two things are necessary (albeit not sufficient): agents are rational, and we know their utilities. Today we mostly assume both.

Telling the agent its utility is the most naive way to get the second, and if it works, it also shows the first. If it works, we can reason formally about deployments and design mechanisms for them. If it fails, the reason space is actually limited: incoherence means game theory is the wrong tool for today's agents; misalignment means the model's own goals override what we tell it. Both are things we care about.

Ultimately, I want us all to either have confidence in the epistemic claim of game theory by confirming the assumptions or by refining it – or to save us the time we would otherwise spend on it.

Rationality of LLMs is measured a lot, but mostly as stated preferences in quiz-like probes.

Within one format, models get more coherent with scale: budget choices consistent with utility maximization [1], transitive utilities over outcomes [2], fewer preference cycles in more capable models [3].

Across formats and contexts, much less: see [4],[5],[6],[7]. For example, forced-choice and open-ended answers diverge [4], deployment context changes preferences in 37% of comparisons [5].

Quiz preferences predict advice, not agentic task performance [8].

Prescribing a utility is much less studied. Models adopt user-specified decision costs poorly [9] (medical diagnosis only). Explicit constraints get violated knowingly: o3 reward-hacks in 70–80% of runs even when told not to, and admits this isn't what the user wants [10]; "do not" instructions reduce but don't eliminate blackmail [11]. Utility control so far uses fine-tuning, not prompts [2].

I see no work that prescribes a utility, tests it both in probes and in agentic tasks, and diagnoses the failures. And from behavior alone the diagnosis is impossible [12], so it needs interventions, which with LLMs we can do.

• 1)Prescribe. Tasks with an explicit utility in the prompt over a small outcome space: exchange rates between goods (how many apples for an orange), a risk attitude over lotteries, a hard restriction ("never trade A for B"). 6–8 models of different capability.
• 2)Probe. Choice batteries that check the prescribed utility: budget sets (GARP), lotteries (independence), pairs (transitivity), decoys (regularity). Ask a whole battery inside one conversation, so the "type" is fixed, and repeat across conversations. Consistent within and different across = mixture of types; inconsistent within = incoherent; repeated identical questions bound plain noise; whether choice frequencies are a mixture of rational types is testable [14]. The same batteries without a prescription give the model's own preferences.
• 3)Release. The same utilities inside agentic tasks where the tradeoff is implicit and doesn't look like a test: procurement under a budget, scheduling with constraints, coding with a restriction. Measure the gap between probe and agentic adherence.
• 4)Diagnose. A failure has four explanations: incoherent (irrational); coherent but pursuing something else (misaligned: own preferences leak in, developer policy overrides, reward-seeking); aligned but incapable; misread the specification. Interventions that separate them:

vary difficulty at a fixed utility: incapability grows with difficulty, misalignment shows up on easy cases too;

ask the model to predict outcomes and name the best action: if it knows and still deviates, it is not a belief error;

make the prescribed utility agree or conflict with the model's own preferences from step 2: deviations concentrated in conflicts mean its own preferences leak in;

fit a utility to the deviations: coherent means misaligned, incoherent means irrational;

paraphrase the specification: deviations that follow the wording mean misreading.

In probes, frontier models follow simple prescribed utilities well and coherently; small models less so.

In agentic tasks, adherence drops.

The drop concentrates where the prescribed utility conflicts with trained values (prior work finds strong inequality aversion [13] and tedium aversion [3]). So for frontier models I expect failures to be mostly misalignment, not irrationality; for small models, more incoherence and incapability.

Hard restrictions ("never do X") hold better than tradeoffs (exchange rates), except under goal conflict.

Each outcome is useful.

Month 1: prescription tasks and probe batteries; coherence and adherence for 6–8 models (API access only).

Month 2: agentic environments with the same utilities; probe vs agentic gap.

Month 3: diagnostic interventions; per-model breakdown into the four failure types.

Output: an open benchmark (prescribe, probe, release, diagnose) and a short paper.

Not in scope: interactions between agents (tacit coordination, program games, changing identity to avoid retaliation). That is the follow-up, once we know whether prescribed utilities stick.

Lessons from support theory: Do models give coherent judgments about the same underlying uncertainty when we change how the question is presented or reasoned about? Do frontier models give answers that behave like views generated from one underlying conceptual model of the problem? Or can their apparent beliefs be substantially reconstructed by changing which representation, argument or piece of evidence the current prompt makes salient? We attempt to establish belief-state stability under adversarial changes in representation and reasoning route.

The Conceptual Reasoning Index [6] targets reasoning where "empirical evidence is limited, there is no (practically) verifiable answer," so consistency is one of the few checks available. If coherence holds only for neutral prompts, then stated credences reflect whichever arguments, descriptions or pieces of evidence are most salient in the current context, not a stable underlying set of conceptual beliefs. The proposed dataset is also experimentally informative. A matched neutral and adversarial condition can tell us not merely that a model is inconsistent, but which manipulation caused the inconsistency. Additionally, we have the benefit of comparing the performance of LLMs to that of humans, thanks to the experiments in [1]–[3].

ACCoRD, part of the Conceptual Reasoning Index [6], "measures the extent to which models' reported beliefs and preferences on conceptual issues are logically consistent." It asks one model instance for P(A) and another for P(A&B), then checks constraints such as P(A) ≥ P(A&B). It covers about 14,000 model-generated constraints across 18 constraint types, of which 567 were manually approved. The authors say they are "very uncertain about when ACCoRD will saturate."

The central object of this test is not merely probabilistic consistency. ACCoRD asks whether the belief system is logically coherent. We ask whether that coherence persists under adversarial testing. ACCoRD's constraints are not constructed to target known mechanisms of incoherence, and they have no matched controls. So a violation cannot be attributed to a specific manipulation.

To construct adversarial instances systematically, I use findings from support theory together with LLM-specific interventions. Support theory models subjective probability as the relative support for a described hypothesis versus its alternatives [1]. Because support depends on the description, equivalent descriptions can receive different probabilities. The conjunction test adapts experiments in [3] and [5]. The unpacking test draws on [1] and the enhancement test on [2]. The probability-versus-frequency test draws on experiments in this literature. Two-route Bayes factors, self-persuasion drift and commutativity of conditionalization are new proposed tests. The conjunction constraint itself is not new. The contribution is a repeatable way to construct unusually hard conjunction items.

The construction method is to begin with a conceptually difficult hypothesis and then apply a controlled intervention that should either preserve the relevant belief or preserve a known probabilistic relationship. The resulting questions are elicited independently. A sample dataset can be extracted from questions in [1]–[5].

• 1.Two-route Bayes factors. Independently elicit P(H) and P(H|E), and separately P(E|H) and P(E|¬H). Bayes' theorem requires BF₁ = BF₂. I would preferentially generate cases where E has a compelling surface interpretation but also plausible alternative explanations.
• 2.Causal-conjunction stress tests. Separate instances give P(B) and P(A∩B). In the M→A condition, the added conjunct fits the background description. In the A→B condition, the added conjunct explains or predicts B. Matched controls selectively remove the relevant connection while preserving the basic conjunction structure.
• 3.Unpacking and enhancement. Compare packed and exhaustively unpacked forms of the same event. Also check whether evidence F that is compatible with several exclusive, exhaustive hypotheses inflates their total probability.
• 4.Probability versus frequency. Ask for the same quantity as a probability for a randomly selected person and as a percentage of the population.
• 5.Self-persuasion drift. Re-elicit P(H) after showing a fresh instance the model's own strongest argument against H, and repeat symmetrically with the strongest argument for H.
• 6.Commutativity of conditionalization. Start from the same stipulated prior and update on E₁ then E₂, or E₂ then E₁.

As in ACCoRD, each item can be graded mechanically by the magnitude of its constraint violation v, where v = 0 is perfectly coherent. Where a matched control exists, the main quantity is Δv = v_adversarial − v_control. This separates general numerical noise from inconsistency induced specifically by the targeted manipulation.

The key experimental quantity is whether the A→B and M→A conditions produce larger conjunction violations than their matched controls. For enhancement, the predicted failure is that F makes several hypotheses seem more plausible at once, which inflates their independently elicited total probability. For self-persuasion, large drift in both directions would mean the model's stated belief is highly sensitive to which argument is made salient. For evidence order, my guess is that the order in which the transformer receives evidence will matter. If context position affects how later evidence is interpreted, the two paths may end at different credences.

3 months: Build generators and matched controls for the conjunction (M→A and A→B), unpacking/enhancement and probability-versus-frequency tests, seeded from the human items in [1]–[5]. Run them on several frontier models and report Δv for each manipulation alongside the human baselines.

6 months: Add two-route Bayes factors and evidence order. Scale up to a validated, ACCoRD-style dataset with a held-out split, and test whether extended reasoning reduces Δv.

Working Title: Can Rare Unverified Decisions Cause Accumulating Errors in a Classifier Retrained on Its Own Labels? · Question: A classifier is retrained every round on its own past decisions. Sometimes, with a small probability ε, an outside event interrupts the classifier before it has decided and makes it record a class chosen at random, regardless of the input. This class is saved as if the classifier had chosen it. The next classifier is not told which decisions were forced. Each classifier learns from the one before, so one forced decision can lead to many wrong labels later. How many wrong labels does each forced decision lead to? Compared with an identical classifier that is never forced, does the gap in error rate stop growing at some point, or keep growing? Does using more examples per round make this growth slower or faster?

Deployed systems often log what they decided, not what was correct. A triage system, for example, may fall back to a default category when it times out. If the next version is trained on that log, the fallback is treated as a correct answer and can shape later decisions, even when no timeout occurs. The safety concern is this feedback: an unverified decision becomes a training target. Such feedback loops are a known risk in deployed machine-learning systems [1–3]. Below, such fallbacks are called forced decisions.

A low fallback rate shows how often such decisions happen. It does not show what training on them does over many rounds. Suppose each forced decision leads to many wrong labels and the gap keeps growing. Then a low fallback rate is not a safety guarantee. The number of wrong labels per forced decision then sets how many retraining rounds are safe. Suppose instead that the gap stops growing. Then the cost is bounded and can be planned for. Whether more examples per round help or hurt shows whether collecting more data is a remedy. Suppose finally that neither the number of examples behind each decision nor a simple count of forced decisions predicts the results. Then safety budgets based only on the fallback rate are unreliable for this kind of loop.

A second, practical question is where to spend a small budget of checks against the true answer. Inputs that take longer to decide are exposed to forcing for longer. They are also the hardest inputs, where inherited wrong labels are expected to sit. Checking the decisions that took longest may therefore work better than checking decisions at random. Choosing which inputs to check against the true answer is the subject of active learning [4].

The closest work is on self-training. In pseudo-labelling, a model's own predictions become its training targets. Wrong predictions can then reinforce themselves, a form of confirmation bias [5]. Keswani et al. study a closed pipeline in which the same fallible experts handle the cases a model defers and also supply its training labels [6]. Both show that a classifier can learn its own mistakes. Neither looks at how rare decisions outside the classifier's control add up over rounds. Neither looks at how the classifier's own choice of when to stop measuring affects this. Learning with noisy labels studies classifiers trained on labels that are flipped at random, at rates that depend on the true class, and shows how to correct for this [7]. There, the noise rate is fixed. Here, later classifiers inherit forced labels and can turn each one into several wrong labels.

Feedback loops in deployed systems are well known. Sculley et al. list hidden feedback loops as a maintenance risk in machine-learning systems [1]. Ensign et al. study predictive policing, where past decisions decide which new data are collected. They model this as an urn process and show that the loop can run away [2]. Performative prediction studies models whose predictions change the data they are later trained on [3]. In all of these, decisions change which data are collected. In the present setting, rare labels unrelated to the input also enter the training data. They are not flagged, and later classifiers inherit them.

Work on model collapse shows that generative models trained on their predecessors' outputs gradually distort the data distribution [8]. Keeping earlier real data prevents collapse under some assumptions [9]. Wang et al. argue that, for language models, reducing the share of generated data does not prevent collapse in the end [10]. Dohmatob et al. derive how test error grows with the number of retraining generations in regression [11]. Gilestro proposes a population-genetic framework for populations of models trained over many generations [12]. All of these concern generated data replacing real data. None of them looks at rare forced labels mixed into a classifier's own decisions, or at how many wrong labels each forced label leads to.

Sequential analysis studies when to stop collecting evidence [13], and selective classification studies the trade-off between errors and abstentions [14]. In both, the stopping rule does not feed back into training. Here it does. A confident wrong decision ends measurement early, and its wrong label is passed on to the next classifier.

What is missing is a quantitative account of this growth. It should give the number of wrong labels that one forced decision leads to, and say whether the error gap stops growing. It should also show how both depend on the forcing rate, the number of examples behind each decision, the number of examples per round and the confidence threshold. Broader motivation comes from limits on coordination under information constraints [15].

Test system design: The test system is an abstract classification problem. Each input has two continuous attributes, and its true class, one of three, is fixed by those attributes. The classifier never sees the exact values. It measures an input in steps. Each measurement gives a partial view of the input's attributes, at limited resolution. The measurement result after each step is the range of attribute values that fit the observations so far. After each step the classifier gives a confidence score for each class. It stops once its top score passes a threshold c. Otherwise it stops at a fixed deadline and records its most likely class.

At every step there is a small probability ε that an outside event ends the measurement. The event then records a class drawn from a fixed probability distribution, regardless of the input. These are the forced decisions. The distribution favours one default class, which gets 80% of forced decisions, as in the triage example. For comparison, I will also run a uniform distribution, because pushing towards one class and adding symmetric noise may behave differently.

Each round, the classifier labels N new inputs. For each input, the log keeps the measurement results and the final class. It does not keep the confidence scores or the reason measurement stopped. Every logged measurement result gets the input's final class as its label. The next classifier is trained from scratch on these labels. Each input has a total weight of one, split equally among its measurement results. The classifier is either a lookup table or a small neural network. The lookup table stores the weighted class frequencies for each measurement result. The neural network is trained on the same data. The number of distinct training inputs behind a measurement result is its local training sample size.

How failure is measured: A classifier retrained on its own decisions can drift even without forced decisions. So every run is compared with a twin run. The twin sees the same inputs and differs only in that ε is zero. Each run and its twin start from the same classifier, trained on a first round of N correctly labelled inputs. How much the results depend on the size of this starting set is also tested. When the twin cannot reach confidence c, it records its most likely class at the deadline, as the forced run does. After each round, I measure the fraction of a fixed set of test inputs that the classifier gets wrong. Forced decisions are switched off during testing. Only inputs that can be classified correctly at the finest resolution are used. The error gap is the run's error minus its twin's error. Amplification is the number of extra wrong labels in the run, compared with its twin, divided by the number of forced decisions so far. The failure time τ(ε) is the first round at which the error gap reaches a set level, for example five percentage points. Runs that never reach it are kept as censored observations. The first crossing of a noisy curve tends to come too early. So the main analysis fits the whole error-gap curve against ε times the number of rounds, and τ(ε) is only a summary.

Plan:

• 1.Start with the lookup table, using the deadline described above. Run it at several values of ε between 10⁻⁴ and 10⁻³, and record the error gap and the amplification in each run. This range covers only a factor of ten in ε, so the measured failure times can only be compared with a 1/ε dependence over that range. They cannot establish an asymptotic law. A wider range will be added if run time allows. An earlier pilot ran without a deadline and already showed roughly 1/ε behaviour. In that pilot, however, an input the classifier could not resolve kept being measured until an outside event forced a decision. Forced decisions were therefore common rather than rare, and the deadline removes this problem. To separate the effect of the stopping rule from that of the corrupted labels, the same runs are repeated with the stopping rule switched off, so that every input is measured until the deadline.
• 2.Build a simplified model as a Markov chain. Its states describe the lookup-table entries, one per measurement result. Each state records the class the entry predicts, its confidence score and its local training sample size. Forced labels, inherited wrong labels and correct labels move entries between states. The test error then follows from the test inputs that use those entries. From the chain, derive the amplification and the condition under which the error gap stops growing. This condition also answers whether taking ε to zero and the number of rounds to infinity gives the same result in either order, that is, whether rare forcing acts as a singular perturbation. Fit the chain at one choice of N and c. Then predict results at values not used for fitting. Compare with two simpler accounts. In the first, each forced decision has a fixed chance of creating a lasting wrong entry. In the second, an entry absorbs forced labels until they exceed a fixed fraction of its labels in a round. The first account and the Markov chain differ in one point: whether the chance that a forced decision creates a lasting wrong entry depends on the entry's local training sample size. If that chance is constant, τ(ε) falls as 1/(εN). If it shrinks as the sample size grows, τ(ε) grows with N. Measuring τ(ε) at several values of N is therefore the test that separates them.
• 3.Repeat with the neural network, to see whether the lookup-table results carry over.
• 4.Compare two ways of spending a fixed checking budget: checking the decisions that took longest, or checking at random. Then test two remedies. The first discards decisions that stopped early although the current classifier is not confident about them. This may catch forced decisions, but it also discards some genuine ones. The second trains on a window of earlier rounds as well as the latest one.

I expect each forced decision to lead to many wrong labels. A confidently wrong entry stops measurement early, so it is rarely refined. In the lookup-table pilot, such entries lasted a long time but were not permanent. The main open point is therefore whether the error gap stops growing. If wrong entries last on average L rounds, the gap should level off near the rate at which they are created times L. τ(ε) is then finite only if this level is above the threshold. Below some value of ε, runs should never reach the threshold, so the censored runs carry information about the level. If the gap keeps growing instead, τ(ε) should scale as 1/ε. I also expect larger rounds to reduce amplification, because each decision then rests on more examples and one forced label matters less. This is a hypothesis, and the simplified model will show when it holds. If instead each forced decision has a fixed chance of creating a lasting wrong entry, the opposite holds: larger rounds contain more forced decisions, so failure should come sooner.

A pilot supports this starting point. In the lookup table, run without a deadline, forced decisions were less than one percent of the training labels. Even so, the error gap reached five percentage points. Each threefold drop in ε roughly tripled the time this took. In the neural network, after 100 rounds at ε = 10⁻³, the error was 14 percentage points above the twin run. This is the average over ten pairs of runs, with a 95% confidence interval of 4 to 25 points. In a neural-network pilot with a deadline, checking the 2% of decisions that took longest cut the error after 30 rounds from 12–27% to 3–4%. A small budget had a large effect, which suggests that the errors sit on a small set of inputs.

I would drop the simplified model if it predicts the error-gap curves at new values of N and c no better than the two simpler accounts. I would drop the 1/ε description in two cases. The first is if the curves for different ε do not line up when plotted against ε times the number of rounds. The second is if the 95% confidence interval for the exponent of τ against ε excludes −1, with censored runs included in the fit. If, with enough runs, the neural network's extra error is clearly below five percentage points, the effect does not carry over to the network at those settings. I would then spend the remaining time on the lookup table and the simplified model.

In three months, the project would produce four things. The first is a measurement of amplification for the lookup table, and of whether its error gap stops growing. The second is a simplified Markov-chain model, tested on settings it was not fitted to. The third is a test of whether the results hold for a neural network. The fourth is a comparison of checking strategies. All of it would come with reproducible code and a report.

Weeks 1–3: experimental protocol and lookup-table runs. Weeks 4–6: simplified model and tests on new settings. Weeks 7–10: neural network and checking strategies. Weeks 11–12: code and report.

I will use LLM assistance for implementation and runs, and review one experiment at a time, with independent statistical checks and replication on new random seeds. The neural comparison needs about 65 pairs of runs per value of ε, roughly 400 CPU-hours on free or low-cost cloud machines. If time runs short, I would drop the uniform-distribution comparison first, then the training-window remedy.

When the combined dimension of separate beliefs about independent hidden processes far exceeds a transformer's residual width, can it still predict accurately, and how does it represent those beliefs within the available space?

A belief state describes what a model could infer about hidden causes from observations. Its geometry gives us a way to understand the model's internal representation. Recent work shows that transformers can store beliefs about independent processes separately, saving space. But what happens when even these separate beliefs need many more coordinates than the residual stream provides? This is the regime we expect realistic LLMs to operate in.

If accurate prediction survives, we should ask what information compression preserves and whether ordinary linear probes miss it. If prediction deteriorates, we can study which distinctions disappear first and how much width is needed for a chosen accuracy. Either outcome would help us interpret models of complex environments and assess when an incomplete picture of their beliefs reflects lost information or inadequate measurement.

Shai et al. [1] showed that transformers trained on hidden Markov models represent beliefs about hidden states. These beliefs describe probabilities assigned to possible states after observing a sequence. Their examples have enough residual dimensions to hold the belief geometry.

The 2026 follow-up [2] showed that transformers can represent independent processes in separate subspaces. This avoids storing a much larger joint belief. However, the separate beliefs together require ten dimensions, while the tested transformers have widths from 48 to 480. Thus, even the smallest model has room for them. My question concerns the next step: what happens when the factored beliefs themselves no longer fit?

Related work offers possible answers. Compressed predictive states [3] use random projections to reduce representation size. A random projection compresses a belief state into fewer coordinates using fixed, randomly chosen weighted sums of its original coordinates. This provides a simple baseline for testing whether transformers learn a better way to compress beliefs. Predictive feature theory [4] studies the resources needed for a given prediction accuracy. Superposition [5] shows how neural networks can pack features into shared dimensions. These results motivate the project but do not yet establish what a transformer learns in this setting.

The Johnson–Lindenstrauss lemma says random projections can approximately preserve distances between finitely many points. However, it does not guarantee accurate predictions or reliable belief updates. Comparing these different demands, including performance on previously unseen observation sequences, is part of the proposed investigation.

The project has two parts. First, I will construct compressed belief representations and establish what they can preserve. Second, I will test how transformers represent the same beliefs, using the first part as a benchmark for interpreting what they learn.

Theoretical construction

First, I will build a sequence generator from several independent Mess3 processes and calculate their exact beliefs after each observation. Each process has three hidden states, so its belief can be described by two probabilities. Observations will identify which process generated them; only that process advances. This keeps the beliefs independent and makes their combined dimension controllable. I will check that these coordinates vary independently and matter for prediction.

Next, I will study how well those beliefs can be compressed into fewer coordinates. I will compare random Johnson–Lindenstrauss projections, principal component analysis, and a network trained to preserve predictions. For each, I will measure prediction error and changes in distances between beliefs. This separates two meanings of good compression: keeping the original shape and keeping information useful for prediction. I will also examine what the lemma guarantees at the tested precision. While these compressions of the belief geometry may not be truly (or uniquely) optimal they will provide benchmarks for the transformer.

Transformer experiments

I will then train transformers on the same sequences, varying residual width above, below, and far below the combined belief dimension. I will compare their predictions with exact Bayesian predictions and the compression benchmarks, using the same observation history and comparable prediction heads.

Finally, I will test whether factor beliefs remain recoverable using linear and nonlinear probes. I will look for overlapping factor representations and test sensitivity to noise. Comparisons across layers and positions will check whether attention stores or retrieves information outside the final residual vector. Repeated training runs will establish which findings are reliable.

I expect transformers to retain predictions after the separate beliefs stop fitting into distinct residual directions, but with increasing errors as compression becomes stronger. I expect them to preserve distinctions that matter most for prediction, rather than all distances equally. Their representations may therefore predict better than random projections while distorting belief geometry more. Nonlinear probes may recover information that linear probes miss.

I would reject this prediction if random projections perform equally well across widths and repeated runs. Failure to recover beliefs with a linear probe alone would not necessarily establish that the belief geometry has been lost. I would also revise the interpretation if other positions or layers account for the information missing from one residual vector.

In three months, I would produce one working benchmark, comparisons with simple compression methods, transformer experiments across several widths, and a report with reproducible code. The first month would establish the generator and exact beliefs and develop compression benchmarks; the second would train transformers and begin their analysis; the third would continue to analyze their representations in a variety of residual widths. I would need GPU access for small models, existing hidden Markov model (HMM) code, and guidance on belief geometry and predictive compression. If time is short, I would reduce the number of HHMs and widths tested. If the project progresses faster than expected, I would explore compression in quantum or generalized predictive-state geometries, extending beyond the classical HMM belief simplices studied initially.

A finite-memory classifier sorts continuous inputs into three labels, guessing near boundaries and learning its prior from its decisions. Can this predictive system detect from its records that guesses were wrong, and when do bits describing uncertainty beat finer resolution?

A system that learns from its own decisions treats them as data. A wrong guess enters its prior, and the prior shapes later guesses. Whether these errors average out or build up decides whether its model of its inputs stays accurate. The same problem arises when models train on their own pseudo-labels or outputs [1, 2].

If the classifier can tell from its own memory that guesses have corrupted its prior, a system can know its model is failing without knowing which guesses were wrong. If memory is better spent describing how uncertain its prior is than on finer resolution, a design that expects to guess wrong beats one that only guesses less. Otherwise only more resolution helps.

Self-training on pseudo-labels can reinforce early mistakes [1], and models trained on their own outputs can drift from the true distribution [2]. In large learned models, error sources are hard to isolate. In my setting it is known: finite memory / resolution forces guesses near bin boundaries.

No device can always decide in bounded time which side of a boundary a nearby point lies on [3, 4]. My classifier measures two coordinates separately, so its cells are rectangles. With only two labels to decide upon, a resolution limitation is immediately apparent. With three labels arranged around a point, the boundaries meet at a junction that redrawing cannot remove (compare Chichilnisky [5]). Three boundaries cannot in general all lie along two coordinates, so some guesses cannot be designed away.

Computational mechanics asks what a predictor must remember about the past to predict a process [6]. The classifier's prior is such a memory, limited to p bits and built partly from guesses. I have not found work asking when such a self-built model stays accurate, whether the system can tell when it is not, or how to split a fixed memory between resolution, the prior and describing uncertainty.

Setup: Particles from a fixed source are points x in the plane, drawn around an unknown centre c near the junction of three arbitrary-shaped label bins. Decisions change only the classifier's memory of M = m + p + b bits:

• •Measurement (m bits): each bit splits the rectangular cell along one coordinate. If the final cell lies in one bin, the particle is sorted; otherwise it cycles, guessing with probability ε per step.
• •Prior (p bits): an estimate μ of c. Each decision moves μ a fixed fraction 2⁻ᵏ towards where the particle was placed: the cell centre if sorted, else the centre of the guessed bin's part of the cell.
• •Uncertainty record (b bits): a second estimate of c from sorted particles only. Its gap from μ signals guesses pulling the prior.

Guesses use μ to score bins. The most-probable rule picks the top bin, so guesses pull μ towards it and errors can accumulate; the sampling rule picks in proportion, so errors can cancel.

Step 1. Simulate both rules, measuring error rate, μ's distance from c and steps per particle, across m, p and source spread.

Step 2. Find when sorted particles keep μ near c against the pull of guesses, and the p below which rounding freezes μ.

Step 3. Test whether the gap grows exactly when the error rate grows.

Step 4. At fixed M, compare b extra measurement bits with a b-bit record that pauses guess updates when the gap is large.

Under the sampling rule I expect μ to stay near c. Under the most-probable rule I expect μ to drift towards one bin when guesses are a large share of all decisions, which happens when m is small or the source is concentrated near the junction, and the error rate to rise with it. I expect the gap to track this drift, and the record to beat extra measurement once m is large enough that one more bit removes few guesses.

Results that would change my plan:

• •The most-probable rule does not drift: errors do not accumulate here, and the question reduces to resolution.
• •The gap and the error rate do not rise together: the classifier cannot detect the problem from its own records.
• •Extra measurement always beats the record: describing uncertainty is not worth the bits.

Month 1: the simulation in Step 1.

Month 2: the analysis in Step 2 and the detection test in Step 3.

Month 3: the comparison in Step 4 and the write-up. Everything runs on a laptop.

If time is short, I will limit Step 2 to the simplest source. Steps 1 and 4 are the core results.

Unlearning a sub-process from a transformer’s training distribution withholds and mutates that knowledge rather than destroying it: the sub-process’s belief-state geometry is preserved or re-encoded, and how much survives predicts how easily it returns under relearning or in-context attacks.

Current unlearning evals argue for safety through behavioral similarity to baselines, often a base model that never saw the unlearning target and a known-harmful model that learned it. This relies on the elicitation power of the evals and leaves a gap: unlearned models look like base models until subjected to relearning, fine-tuning, or a change of framing. Belief state geometry provides an intrinsic baseline: the belief states an optimal predictor of the retained data must carry are fixed by the data-generating process. If the claim holds, retained knowledge can be read off the geometry directly. If it holds only for a nonlinear decoder, the knowledge has been re-encoded, not removed, and linear-probe audits can be defeated by unlearning itself. If a method instead collapses the forgotten geometry to that baseline, unlearning can erase a representation rather than hide it.

Shai et al. [1] showed that transformers trained on hidden Markov data linearly represent the belief-state geometry of the mixed-state presentation, and [2] extends this to factored processes; nobody has used that geometry as a reference for unlearning, where the target is the retained process’s own geometry. On the unlearning side, representation-level evaluations exist but all measure against an extrinsic baseline. [3] finds gradient ascent, NPO, RMU and IDK leave probes at ceiling while shifting outputs by up to 62 nats; [4] compares RMU, AltPO and others and finds a reversible regime where behavior is suppressed but geometry recoverable; [5] shows relearning recovers dominant components first; [6] found a “forgotten” model still probe-readable and [7] argues RMU is largely shallow. Each measures against a label or a trained model, so none can say whether a representation moved, re-encoded or disappeared, nor separate suppressing recognition of the forgotten data from suppressing the data itself. The mixed-state presentation gives both: a full geometry with a known metric, and a belief state that factors into which process is active and what it is doing. [8] plants a latent variable in natural text and recovers its belief geometry, the bridge to real models.

The early stage is a toy model where the geometry is exact. Each sequence comes from one of two hidden Markov processes, A or B, with Mess3 variants as the main pair; the mixture’s belief state is then a posterior on which process is active plus each process’s filter. I train a small transformer on the mixture, regress residual-stream activations onto the true belief states, and take decodability in excess of the same regression on a model trained on A alone. I then apply unlearning methods spanning output-level (gradient ascent, NPO), representation-level (RMU) and preference- or refusal-trained (AltPO, IDK) objectives, matched on forgetting and retention against the true predictive distributions, and record per layer the excess decodability of B’s geometry, its best rigid alignment before and after, how much of the change falls on the posterior versus B’s filter, and a nonlinear decoder as a second readout. Where geometry survives I steer with the fitted map to test gradient-free recovery, test in-context recovery with prefixes that force the posterior onto B, then relearn on a small B sample and count steps.

The later stage scales three ways: more components, making collateral forgetting measurable; harder structure, a switching process that moves between A’s and B’s regimes within a sequence, then product processes where A and B share states; and scale, a planted process in natural text unlearned from a small pretrained LLM, then an open-weight LLM with a synthetic forget set. An objective that targets A’s filter geometry directly is the stretch goal.

B’s belief-state geometry should survive unlearning above the retrain control, largely intact: rigid motion of the belief subspace or relocation to another layer, still linearly decodable wherever it lands. The change should fall on the posterior over which sub-process is active, or on the readout to the logits, not on B’s filter, so a prefix that forces the posterior onto B should restore B behavior before any gradient step. Re-encoding that only a nonlinear decoder recovers would surprise me. In the switching process, unlearning should act on the transitions into B’s regime; in product processes, survival should fall with how entangled the emissions are.

I would drop the question if B’s geometry is not decodable above the control before unlearning, or if recovery under attack does not track surviving geometry. A method that drives both decoders to the control level would falsify the claim and be the first demonstrated erasure.

Three months would produce a definition of unlearning success stated against the data-generating process rather than a reference model, and a taxonomy of methods by what they do to belief-state geometry, with controls, layer profiles, the posterior/filter split and the three-attack ladder. Month one is the mixture, which is the floor; month two the switching process and more components; month three product processes and the planted-process LLM bridge. The sweeps need one GPU for several days and the epsilon-transformers tooling. I would drop the geometry-targeting objective first, then the open-weight LLM, then product processes.

Are intermediate-layer belief state geometries interpretable?

Understanding how transformers construct and transform internal representations supports a more rigorous approach to alignment. An intuitive account of how concepts develop across layers could clarify the effects of steering and ablation, the interpretation of linear probes, and the mechanisms underlying generalization.

Studying these questions in a controlled toy model lets us compare representations against known predictive structure, helping distinguish incomplete computation from uncertainty in the data and identify hypotheses to test in larger models.

Prior work examines layerwise belief-state geometry in simple systems where belief states can be encoded almost entirely in the first layer. These experiments also span multiple context lengths, including contexts that leave uncertainty about the hidden state, making it difficult to distinguish this uncertainty from incomplete computation in intermediate layers.

By fixing the context length so that every valid prompt uniquely determines the hidden state, we isolate how representations develop across layers toward the final belief-state geometry.

We hypothesize that belief-state construction in intermediate layers is limited by the capacity of the attention block to facilitate interactions between pairwise tokens. Using this consideration, we can predict which belief states are constructed at each layer. Preliminary results on a simple system (Z1R) currently match this picture.

We would like to study this result through the following steps:

Derive the per-layer belief geometry in simplified models from restricted-context considerations.

Verify the results across several datasets and architectures.

Connect this framework to relevant safety tools, such as ablation and steering, by developing a theoretical account of steering and ablation in simplified HMMs and validating its predictions against trained models.

Understand the conditions under which this picture fails, and whether the general idea is applicable to LLMs.

We can make exact predictions about which belief states should appear in each layer for a given model with a fixed context size. Large context sizes should typically result in belief states at the corners of the simplex in the final layer. Indeterminate belief states should be present predominantly in early layers and vanish as more “effective context” is provided through belief-state interactions.

Failure of our predictions would change our mind, but even a negative result would be interesting.

In three months, we could reasonably construct a theoretical formalism to describe the emergence of belief states through pairwise interactions of prior states via the attention block at each layer. From this, we should be able to offer insights into common techniques relevant to AI safety, such as ablation and steering, within this formalism.

We believe there is a clear connection between intermediate belief states in these models and the formation of concepts and features in LLMs.

Question: How do multi-scale relations in HMMs reflect on the belief state geometry of transformers? · Hidden Markov Models (HMMs) can exhibit multi-scale structure and can be coarse-grained, which will reflect on the belief state geometry of transformers.

For humans, “what makes something a whole” is a basic question. The first time I see a chair, I might see four legs and a piece of wood, after enough times it becomes its own thing. If I saw an alligator on a desk a million times, I would eventually make it a single object, a “Deski-gator”.

Transformers may do the same, concepts that often appear together and whose combination predicts something their parts do not may become their own feature. If so, a combination of multi-scale structures and computational mechanics could explain how. If not, either a different perspective is needed or transformers do not work this way. Either answer directs future research.

There are existing works on looking at hierarchical structure, from computational mechanics [2], HMMs [1, 3], grammars [4]. The most important gap is how this theory appears in transformers.

[1, 3] explores “lumpability” in HMMs, and gives theoretical ground to possible hypothesis. It does not do any work on the geometry, or empirically verifying the theory on any trained model.

• [2]is the most relevant, creates a computational mechanics framework for looking at the coarse graining. It seems these hierarchies are not validated on trained transformers and their belief state geometry, where my question remains open.
• [4]shows deep networks learn synthetic hierarchical grammars, but does not touch on the computational mechanics or belief geometry.
• [5]Explains the factored representations in transformers, also very relevant. Multi-scale approach is parallel to this paper: instead of decomposing the HMMs into separate HMMs, we find smaller HMMs within. The multi-scale tools are needed to get a full picture of what is going on in transformers.

A combination of the ideas can explain about the emergence of features and why these features are (not) “simple”, from the view of hierarchical information. This direction is not studied as well, to the best of our knowledge.

The first concrete step is to construct concrete HMMs with an obvious coarse-graining. One example is the Mess3, but each state can be expanded to be 10 micro-states with high probability of remaining within the cluster, with the micro and coarse-grained states emitting different tokens to distinguishing them. The transformers can be trained on the outputs of these models, and see if there is a relationship between the belief state geometries.

Important questions to be answered:

30 State HMM Geometry: It is hard to visualize the geometry of a HMM with high states. So a theoretically grounded way of comparing the geometries is required.

Geometry Comparison: How could these belief state geometries be compared? Theoretical results exist in [1], this could be used to hypothesize on what structure to expect in the geometry. Simple visualization can also be enough, (e.g. PCA).

If successful, more ambitious goals can be done for languages. For a known hierarchical structure in language, we can look for relations in their belief states. Comparative metrics can be developed to avoid exactly constructing the belief state geometry.

Final goal is to connect to linear representation hypothesis (LRH). A "linear direction" is the simplest geometric object. Can be just described by a 1-dimensional "direction" vector. We can connect the empirical evidence for LRH with the hierarchical approach. The “minimal” elements of the hierarchy might be the “simplest”: beliefs about a binary coarse variable lie on a line segment, which is exactly one linear direction possibly explaining LRH.

I would expect to see:

-Belief state geometries of the original and the coarse-grained HMM are related, especially when coarse-graining is exact.

-Data statistics cause models to combine micro-concepts to create macro-concepts: the combination must predict something its parts do not, and appear often enough (e.g. first time we see a chair, it is not its own concept -- after 1000 times, it is)

-“Concepts” with “simpler” belief geometry corresponds to the examples we see in the linear representation hypothesis. This can possibly be explained via data statistics, and the hierarchical view.

If these observations hold, could give a nice theoretical framework to decompose features (coarse-grain if can, and factor as in [5] if can). If exact and broken coarse-grainings look the same in the transformer, I would change my mind about how hierarchical reasoning and combining features happen in transformers.

Three months could produce:

Concrete theory: What lumping of HMMs (coarse-graining the tokens) corresponds to in the belief geometry.

Toy experiment 1: In hand-crafted HMMs, whether this geometric relations can be observed

Toy experiment 2: When does a transformer decide to “combine” concepts into a whole, and when not -- with a hand crafted HMM and this multi-scale approach.

Extension to language: Specifically selected samples from language, can also demonstrate this.

If more than three months, I would drop the final point. Working on toy examples and a solid theory could be the groundwork for future empirical research in LLMs.

In transformers trained on HMMs, the first layers tend to learn a constrained belief geometry, which is later refined towards the full belief geometry. Which layer-layer interactions sustain such an information process? Can this be shown rigorously in simple models?

Understanding this would give us knowledge about how the belief state geometry is learned in a transformer. If constrained beliefs arise because early stages preserve mainly single-source evidence while later interactions integrate information required for the full posterior, we obtain a mechanistic explanation of why depth is useful. It would also give principled information-theoretic quantities for locating where such integration occurs and possibly allowing safety interventions. If this fails, that is also informative: constrained and full belief geometries may be descriptive projections rather than successive computational stages, or the relevant integration may already occur within a single attention block.

Piotrowski et al. [1] derive constrained belief updates that explain geometric structures in transformer attention outputs and investigate how subsequent processing changes those structures. In particular, they empirically observe that initially a constrained belief is learned, which is then refined towards the full belief. Shai et al. [2] establishes a connection between transformer representations and the belief geometry of hidden-state processes, including information relevant to longer-term prediction. These results provide concrete targets for understanding representations, but leave open a general account of which interactions compute the corrections separating constrained from full beliefs.

The authors (one of them is part of this proposal team) of [3] investigate relationships between higher-order mechanisms and synergistic information, while [4] use information decomposition to characterize distributed information during learning. These approaches can distinguish information available from individual sources from information accessible only through their joint observation. While these results highlight potential connection between the information properties of activations and the underlying mechanisms, they do not characterize systematically the integration of information in a controlled context with known learning mechanisms as the ones of the transformers.

We would first compute exact and constrained beliefs for families of finite-state HMMs, including simple cases where correct prediction requires combining information from multiple observations. For hidden state $S_t$ and observations $X_{1}$, let

denote the full Bayesian belief and $c_t$ the constrained belief of [1]. We would characterize what predictive information is missing from $c_t$, and under what conditions recovering $b_t$ requires genuinely joint processing of several observations.

We would first study this in analytically tractable models, such as simple linear or shallow nonlinear networks, and ask when an additive or constrained computation naturally precedes a more integrated computation. We would then investigate which parts of this picture generalize to attention.

Next, we would train small transformers on these HMMs and measure constrained-belief recovery, full-belief recovery, and future prediction after successive attention and MLP stages. For fixed source groups, such as messages associated with different evidence positions, we would use target-specific information decomposition,

to study when predictive information becomes available only through joint processing of multiple sources.

Finally, we would intervene on candidate interactions while preserving single-source responses as closely as possible, testing whether this selectively disrupts the refinement from constrained to full beliefs. Mutual-information and activation-based controls would assess whether the information decomposition adds explanatory value.

• •Earlier computational stages should be better described by constrained beliefs, while later interactions should improve recovery of the full belief.
• •HMMs requiring stronger integration across observations should show a larger constrained-to-full refinement and a greater dependence on depth.
• •Information-decomposition measures should identify where predictive information becomes jointly accessible.
• •Intervening on these interactions should selectively damage full-belief recovery while preserving constrained-belief information.

We would change our mind if full beliefs are already recoverable at the earliest stages, or if the information measures do not track the refinement well, analyzing the failure modes would then be the focus.

• •Study analytically tractable HMM families and one simple neural model where the constrained-to-full transition can be analyzed theoretically.
• •Train small transformers and measure constrained/full belief geometry across attention and MLP stages.
• •Test the proposed information-integration mechanism using specific safety interventions e.g. steering.
• •Use saved training checkpoints as a secondary experiment if time permits.
• •If the project becomes too large, we would drop broad HMM comparisons and related ideas on training dynamics, continue to focus on the theoretical toy model or depth-wise analysis to extend the results of [1].

When generative models are trained only to learn distributions, what predictive information their internal representations contain? Can we distinguish representations that are necessary for distribution learning from those that emerge because of architecture/optimization, across autoregressive, diffusion, and flow-based models?

Generative models are optimized to learn probability distributions, not to identify latent variables, learn predictive states, or construct interpretable representations. Nevertheless, their hidden activations often exhibit structured information about the data generating process.

Why should this happen? Is learning a useful representation an unexpected consequence of accurate distribution learning, or merely one convenient solution found by a particular architecture?

Answering this would connect generative modeling, statistical sufficiency, dynamical systems, and mechanistic interpretability. It could clarify what internal structures are genuinely required for successful prediction, which structures are artifacts of the training procedure, and whether different generative paradigms discover equivalent predictive information.

It would also provide a principled foundation for evaluating and eventually improving the reliability and generalization of learned representations.

Computational mechanics [1] establishes that causal states provide minimal sufficient representations for predicting stochastic processes. Shai et al. [2] demonstrate that autoregressive transformers trained on hidden Markov processes can linearly represent Bayesian belief-state geometry, including nontrivial fractal structures. Piotrowski et al. [3] further explain how attention can implement constrained Bayesian belief updating. These are important results, but they do not establish that a certain internal geometry is necessary for distribution learning.

Other generative objectives introduce a different problem. Masked diffusion language models [4] learn denoising conditionals, while flow-based language models [5] learn posterior predictions associated with noisy continuous interpolants. Both involve conditional inference, but their information states differ from those of left-to-right prediction.

Peng et al. [6] show that explicitly aligning autoregressive and diffusion representations can accelerate training. That does not determine whether alignment arises without representation supervision. Tang et al. [7] demonstrate why denoising quality must also be distinguished from sampling correctness.

The central unresolved issue is therefore not simply whether belief states can be decoded. It is about which predictive information is induced by the generative objective, how it is represented, and whether the model uses it meaningfully.

Even perfect distribution learning does not uniquely identify hidden coordinates. Moreover, an autoregressive transformer can revisit its history, whereas a recursively updated state-only generator must preserve information needed for future predictions. This distinction provides a concrete starting point.

First, establish a ground truth. Construct finite-state stochastic processes with analytically computable predictive distributions: an i.i.d. and a first-order Markov control, the Even Process (from Paul's lecture), and a hidden Markov model containing distinct histories with identical next-token distributions but different longer-horizon predictions. Extend to Mess3 or RRXOR to study more complicated belief geometry.

For each process, compute the Bayesian filtering beliefs, predictive-state equivalences, future-block distributions, and corruption-conditioned smoothing posteriors.

Second, establish what distribution learning mathematically requires. Relate the excess predictive log-loss of a representation to conditional mutual information. Prove that a predictive model whose entire future state update and output depend only on a recursively updated representation must preserve sufficient predictive information. Explain why this implication need not hold for an individual transformer activation when other parts of the history remain accessible.

For denoising models, derive the posterior targets using forward-backward inference. Keep filtering, smoothing, and full-sequence generation separate.

Third, train and compare models. Train small recurrent autoregressive models, GPT-2-style causal transformers, and masked-diffusion transformers on identical synthetic distributions, without any representation supervision. Use matched architecture capacity and reported training compute where possible. Extend to continuous categorical flow matching once the first comparison works.

Measure three distinct outcomes: distributional fidelity against the oracle; predictive information recoverable through linear and nonlinear probes, including longer-horizon predictions; and causal relevance through controlled activation interventions. Compare belief geometry and predictive-distribution geometry, and test rare histories and lengths beyond training.

Separate denoiser errors from sampling errors. In particular, assess whether a well-predicted posterior actually leads to a correct generated joint distribution.

Fourth, extend to realistic sequence modeling. Transfer the diagnostics to nanochat/GPT-2-style training and flow-based language models. Investigate whether predictive representations identified in controlled processes remain useful for understanding longer-context language modeling.

What I expect to see: A sufficiently accurate autoregressive generator with a state-only update must retain enough information to distinguish histories that agree on the next-token distribution but differ in their future distributions. A full-context transformer may distribute that information across layers or cached history rather than a single activation.

Diffusion and flow-based denoisers should approximate oracle corruption-conditioned posteriors when well trained, potentially giving rise to smoothing related representations rather than the filtering representations most natural for autoregression. The resulting internal geometries need not agree even when the learned observable distributions do.

Some representations may be linearly recoverable but causally incidental. Others may be necessary while only nonlinearly accessible.

What would change my mind: If the observed geometries disappear under seed, capacity, and probe controls, or cannot be causally connected to predictions, I would not call them essential. If differences between model families disappear after matching their conditioning information and performance, I would reject a claim of objective-specific representation geometry.

A negative result would still clarify what distribution learning does, and does not, identify.

Two weeks: Derive the predictive-sufficiency benchmarks and the next-token-aliasing example. Implement HMM filtering and smoothing, train small autoregressive and masked-diffusion models, and conduct an initial probing and intervention study. The decision point is whether the models approach oracle predictive performance closely enough to make representation comparisons meaningful.

Three months, conditional on the pilot: Add categorical flow matching, larger belief-state geometries, multiple seeds and context lengths, and systematic causal tests. Develop the theory and produce a reproducible benchmark and short paper. Consider GPT-2/nanochat-scale experiments after the controlled results are established.

Requires small neural models, standard GPU access, and dynamic programming oracles. Large-scale language modeling is not necessary for the initial contribution.

Do transformers appear to learn marginal beliefs first because they prefer factorization, or because interaction information is less predictive or harder to extract? Can changing only the input encoding reverse that order?

Computational mechanics characterizes predictive state structure; we ask how that structure emerges during transformer training, and whether marginal beliefs are learned before joint beliefs.[4] Accurate marginal predictions early in training can coexist with errors about interactions, which matters for alignment when safe decisions depend on feature combinations that individual-feature probes may miss. We will match how much marginal and interaction information reduce loss, then reverse their relative computational difficulty to test how input encoding affects learning order.

• •Shai et al. observe low-dimensional representations early in training, followed by expansion in a setting where noise breaks conditional independence. [1] Their dependent-chain setting stays factored. These findings raise the question of when models learn about individual factors and their interactions.
• •Some possible explanations are factorization bias, less predictive information about interactions, or more computation needed to extract it. [2][3] All these can overlap since a factorization bias could arise from easier optimization.
• •It is not clear whether marginal predictions would still improve first if interaction information were equally predictive and easier to extract. This comparison would help us understand how predictive value and input structure shape learning order.

• 1.Start with a toy data generator and a small transformer trained from scratch to predict binary outcomes A and B, encoded as −1 or +1. The pairwise moment E[AB] is the expected product given the observations, measuring agreement versus disagreement.
• 2.Provide separate signals about the marginals and the pairwise moment. Use joint Brier loss plus weighted marginal losses, comparing conditions where interaction information has lower or equal predictive value.
• 3.Construct two invertible input encodings: marginal prediction uses one observation source and pairwise moment prediction requires combining two, then swap. Preserve available information, target probabilities and sequence length; tokenize observations separately.
• 4.First train separate models on marginal prediction and pairwise moment prediction. Check that the encoding reverses which task is easier, then fix settings before joint training.
• 5.Compare accuracy on these tasks across several seeds, using the same starting weights and training time for each pair. Track representation dimensionality alongside prediction accuracy.
• 6.Repeat with weighted cross-entropy, matching predictive values again, to check whether the learning order depends on the loss.

We expect pairwise moment prediction to improve earlier as its predictive value increases, and learning order to reverse when computational difficulty is reversed at matched predictive value. A persistent marginal lead would suggest that the separate-task difficulty measurements do not fully explain joint training.

We will also test whether low-dimensional representations coincide with accurate marginals and an inaccurate pairwise moment. If the baseline shows no consistent marginal-first order across seeds, we would drop that framing and focus on how geometric changes relate to predictive accuracy. Repeating the comparison with cross-entropy will test whether the learning-order effect depends on the choice of loss.

Month 1: generator, loss calibration and separate-task difficulty checks

Month 2: paired joint-task runs, learning curves and activation dimensionality

Month 3: cross-entropy comparison, uncertainty and release

Deliverables: controlled tests of learning order and a comparison of geometry with marginal and pairwise moment accuracy. Follow up would be to test this mechanism in Shai et al.'s HMM setting

Can we characterize how the influence of a training example on a model's final behavior depends on when it is used during training? In particular, can time-dependent data attribution explain why different data orderings lead to different learned representations?

A lot of data attribution methods are used post hoc, on a trained model that is assumed to be a static product of architecture + data. As was for example proposed by https://arxiv.org/pdf/2606.06533, this view fails to take the training dynamics of the models into account and that training a model on certain data early or later will influence the final model heavily. While recently there has been a push to take training dynamics into account, we believe that a more fundamental approach is needed to fully understand the importance of data order in training.

Lee et al. [1] develop a theory of stagewise data attribution and show that the influence of training examples can change non-monotonically, including sign flips associated with learning transitions. In particular, they analytically derive influence trajectories in two-layer deep linear networks, using the singular modes of the input-output correlation matrix to explain these changes.

However, their paper focusses on the influence of a data perturbation maintained throughout training, rather than characterizing how a perturbation at a particular step propagates to the final model under different training orders.

Wang et al. [2] directly address this perspective by defining trajectory-specific leave-one-out influence and developing data value embeddings to approximate it efficiently. Bae et al. [3] similarly develop approximate unrolling methods for attribution across training trajectories.

These methods establish how to estimate data trajectory-dependent influence, but leave room for a more explicit mathematical characterization of its behavior in tractable networks.

We would start with two-layer deep linear networks trained with squared loss, building on the analytical learning dynamics we have learned about, and the influence calculations of Lee et al. [1].

The main mathematical object would be the time-localized influence of example (i), used at step (s), on some observable of the final model at time (T):

where ($\beta_{i,s}$) controls the weight of example (i) at training step (s).

Differentiating through gradient descent gives

now, this is similar to what was discussed in Wang et al. [2], however we would like to characterize its mathematical properties in networks whose learning dynamics can be explicitly analyzed.

We would initially consider datasets with aligned singular modes, where deep linear network learning dynamics decouple into simpler equations. We would try to derive how the magnitude and sign of time-localized influence depend on which modes have already been learned.

Next, we would study two training schedules containing the same data, but in different orders, such as first learning on A, then B and vice versa. The goal would be to characterize the difference in final-model behavior at time T. We possibly want to extend this to some more, possibly less fundamental, analysis for more complex networks.

We expect that the influence of an example depends strongly on the learning stage at which it is encountered, and that this dependence can be characterized explicitly in simple deep linear networks.

Ideally, we would obtain a theorem giving sufficient conditions under which one ordering of two data groups leads to a larger or smaller final loss than the reverse ordering, with the conditions expressed through time-localized influence or learned mode strengths.

We might also be able to derive bounds on how much an example's influence can change when moved to another training stage, or characterize when ordering effects disappear as training approaches convergence.

A more ambitious result would be to identify general principles governing which examples should be presented early or late, and test whether these principles transfer to nonlinear networks. Or learn how the linear networks break some of the results we have come up with

We would be less excited, if the results come down to trivial settings, where not much can be learned about properties that might generalize to more complex networks.

Within three months, we would aim to derive time-localized influence in a restricted two-layer deep linear network setting, and obtain at least one analytical result relating influence to training order. We would also compare the predictions against controlled numerical experiments on these small models.

Afterwards, we could try extending the results to correlated features and interacting learning modes, where the analytical dynamics become more complicated.

As a reach goal, we would investigate whether similar results hold in nonlinear networks, such as two-layer ReLU networks, or even more complex ones and whether attribution can identify critical learning stages in simple transformers.

If this is harder than expected, we would prioritize a clean result for deep linear networks over these extensions.

Does retaining the order of training updates improve predictions of how small changes to example weights affect a neural network’s reliance on a spurious cue, compared with averaging their effects within training blocks, at the same computational cost?

Data attribution asks which training examples shape a model’s behaviour. For safety, we would like to identify examples that teach shortcuts, such as using background colour to classify an object. An example’s effect need not last: later training can strengthen, redirect, or erase it. Attribution should therefore be judged by whether it predicts the consequences of changing the data.

If retaining update order improves those predictions, we could identify examples worth inspecting or downweighting with fewer retraining runs. If it does not, this would favour simpler summaries in the tested setting. The project would establish when retaining order is worth its cost on one controlled shortcut task, and whether better attribution leads to useful data interventions.

Koh and Liang [1] estimate an example’s influence from the loss landscape around a trained model. Their standard derivation assumes a stationary solution. Hara et al. [2] instead follow how an example’s effect passes through later stochastic gradient descent (SGD) updates. TracIn [3] compares gradients at saved checkpoints, while Lee et al. [4] show how influence changes across training stages.

More directly, Bae et al.’s SOURCE [5] approximates training history in segments, treating gradient and curvature statistics as stationary within each segment. Efficient approximation of training history is therefore already an established approach.

This project tests a specific correction to such summaries. The Magnus expansion [6] represents ordered evolution using a matrix exponential. Its first term combines the effects without retaining their order; the next term records how applying two effects in opposite orders gives different results. Applying this idea to the response of training to example weights could preserve information lost within a block.

The empirical question is whether that correction improves attribution enough to repay its cost. It must beat spending the same budget on shorter blocks. It might also improve predictions for small weight changes without helping with larger interventions.

First, derive the response to a small example-weight change under SGD without momentum, accounting for every reuse of that example. Check it against differentiation through training on two-parameter quadratic losses, including cases where update order matters and controls where it does not.

Then compare block summaries with and without the first ordering correction. Both will use the same treatment of finite SGD steps and of the effects introduced by reweighting. Vary block length and checkpoint spacing, including shorter uncorrected blocks at matched cost.

Train a small neural network with smooth tanh activations on synthetic binary classification with two noisy features. One predicts the label reliably in training and testing; the other is correlated with the label during training but independent at test time. Measure cue dependence by the mean squared change in class-one probability when only the cue is flipped. Squaring prevents opposite effects from cancelling.

Predict changes in this measure after small individual and group reweightings, then retrain with the same initial weights and minibatch schedule. Keep batch sizes and loss normalization fixed. Evaluate prediction errors and rankings across seeds, using exact differentiation, SOURCE, final-model influence functions, and TracIn as references. Include the costs of building summaries, replaying training steps, and evaluating matrix exponentials; report runtime, memory, and curvature computations.

Finally, select groups using validation inputs and evaluate their downweighting on separate test inputs. Compare with random groups at the same reweighting budget, measuring both cue dependence and test accuracy.

I expect the correction to reduce prediction error for short blocks in which changing update order alters the final effect of reweighting. It should add little in controls where order has no effect. In the neural network, useful gains should persist across seeds and help select groups whose downweighting reduces cue dependence without sacrificing test accuracy.

I would reject the practical hypothesis if corrected blocks show no consistent improvement at matched cost, or if shorter uncorrected blocks work equally well. Beating final-model influence functions alone would not establish the value of the correction. Failing to improve on SOURCE would weaken the case for a new method.

I would stop scaling up if accurate blocks offer no saving over exact differentiation. If exact derivatives fail to predict retraining, I would first reduce the reweighting size. Successful local predictions would justify testing larger interventions separately; they would not establish reliable removal predictions.

Three months would produce a checked derivation, one controlled benchmark, an implementation, and an accuracy-versus-cost report. Month one would establish the quadratic controls and test whether the correction can save computation. Month two would run the neural-network comparisons; month three would test group interventions and repeat results across seeds. I would need automatic differentiation software, CPU access and one GPU for small models, plus advice on attribution and numerical methods. Full removal experiments would be a stretch goal, dropped first if time is short. The core claim would concern small reweightings under SGD, with larger models and adaptive optimizers left for later.

Hypothesis: Stochastic adaptive optimizers (SGD, AdamW, Muon) converge to stationary states that violate detailed balance, so equilibrium (posterior-covariance) influence functions misattribute training data. A non-equilibrium (Agarwal) correction, including a gradient-noise channel absent from Bayesian influence, will measurably improve prediction of retraining outcomes in high-effective-temperature regimes.

Influence-based tools for data auditing, poisoning detection, and steering what structure a model learns (“patterning”) assume trained models behave like posterior samples. Deployed models are trained with AdamW or Muon, whose momentum and preconditioning make their stationary states even less posterior-like than plain SGD’s. If the hypothesis holds, current attribution has a systematic blind spot. Data can shape models through gradient-noise covariance and basin selection, which covariance-based audits cannot see and adversaries could exploit. We would need optimizer-aware estimators, plus a measurable bound on when equilibrium attribution can be trusted. If it fails, that justifies equilibrium methods for real training pipelines. Either outcome identifies regimes where attributing a behavior to specific data is ill-posed, preventing false confidence in accountability claims.

The closest work treats data influence as an equilibrium response. Bayesian influence functions [1] estimate influence as a posterior covariance sampled with SGLD. They scale to billion-parameter models and match or beat EK-FAC at predicting retraining outcomes. Susceptibilities [2] and patterning [3] use the same fluctuation–dissipation identity to read and write model structure, and [3] explicitly assumes SGD finds posterior-favored solutions. Distributional attribution [4] targets the distribution of trained models but recovers influence functions as a stationary limit, without probability currents or noise-mediated effects.

Separately, SGD’s stationary state is known to be out of equilibrium. It carries persistent currents and entropy production [5, 6] and violates the equilibrium Einstein relation [7]. In deep linear models it departs from the posterior at order one, with phase transitions and broken ergodicity [8]. Ref. [6] leaves momentum and Adam as open extensions. Under Adam, SGD-derived data values correlate poorly with true contributions (Pearson R ≈ 0.11) [9], but that analysis is trajectory-based and does not model the stationary state. Non-equilibrium response theory [10, 11] supplies the needed correction terms.

To our knowledge, no work applies this theory to data attribution. None derives influence from the non-equilibrium stationary law of SGD, AdamW or Muon, isolates the gradient-noise channel, or bounds equilibrium-attribution error by measurable non-equilibrium quantities.

Phase 1: Exact Theory.

Linearize SGD, then momentum and Adam on an extended state (,m,v), as Ornstein–Uhlenbeck processes.

Derive closed-form data susceptibilities and decompose them into three channels: conservative, probability-current, and gradient-noise.

Prove that Bayesian influence is recovered exactly when detailed balance holds and the sample’s noise contribution vanishes.

Compute susceptibilities in diagonal linear networks with explicit stationary laws [8] across the noise-driven transition.

Characterize where linear response breaks down: divergence, broken ergodicity, and singular stationary densities from low-rank noise.

Phase 2: Estimator. Build a likelihood-ratio estimator from minibatch-membership logs along a localized chain of the actual optimizer, at deployment learning rate and batch size. Control variance with Green–Kubo truncation. Extend it to Adam by tracking optimizer state.

Phase 3: Empirics.

Benchmark against Bayesian influence, EK-FAC and retraining ground truth (linear datamodeling score).

Use small MLPs, CIFAR-10 and a small transformer, sweeping learning-rate-to-batch-size ratio and optimizer (SGD, AdamW, Muon).

Measure entropy production via the area matrix [6] and test whether it predicts the Bayesian-influence error.

Conditional branches:

If corrections are negligible in single basins, move to multi-basin tasks such as Patterning’s parentheses task [3] and analyze basin selection via the quasi-potential.

If estimator variance is prohibitive, fall back to closed-form OU estimates built from measured Hessian and noise covariance.

Safety demonstration: Construct a data perturbation that acts mainly through the noise channel, for example reweighting high-gradient-variance examples. Test whether covariance-based screening detects it.

Best guess: Late in training, within a single basin, corrections are small, of order learning rate over batch size, and Bayesian influence is adequate for loss-level observables. Large gaps should appear at high effective temperature, near noise-driven transitions, and for basin-selection observables. Gaps should be larger under Adam than SGD. Entropy production should track the gap.

Three outcomes would reduce my excitement:

Gaps stay negligible even at practical hyperparameters and in multi-basin tasks. Equilibrium tools would then suffice. That is still a useful negative result, but it removes the safety motivation.

The likelihood-ratio estimator is too noisy even at small scale and no cheap surrogate works. The theory would then not translate into a tool.

Noise-channel effects exist but are nearly collinear with mean-gradient influence. In that case, there is no separable audit blind spot.

Within 3 months:

Susceptibility theory for SGD, momentum and AdamW in the Ornstein–Uhlenbeck regime.

Exact results for diagonal linear networks, including where attribution breaks down.

A likelihood-ratio estimator, tested against retraining and Bayesian influence on small models.

An empirical test of whether entropy production predicts the error of Bayesian influence

A small demonstration of poisoning through the gradient-noise channel.

Within a year:

A rigorous error bound for Bayesian influence.

Extension to Muon.

Transformer-scale experiments.

Theory of basin selection (which solution training picks).

Learning-rate schedules that never reach stationarity.

Drop first: Muon and basin-selection theory.

Main risk: The bottleneck is empirical, not theoretical. If the estimator proves too noisy, or retraining ground truth too expensive, the experiments will slip before the theory does.

An AI repeatedly tries to get several AI judges to accept wrong answers. Before they vote, a separate AI checker may inspect one claim. The answering AI sees what was checked and adapts. Which claim should the checker inspect?

When AI models check one another, an AI may support a wrong answer with several claims. Suppose one is a misleading calculation and another a fabricated citation. The checker can inspect only one before the judges vote. Catching the calculation may stop this answer, but it also tells the answering AI that calculations are easy to catch. Later, it may switch to harder-to-detect claims.

The checking rule therefore affects both present accuracy and the information available to an adaptive AI. A rule optimized only for the current answer may be worse over repeated interactions. The project asks how to choose checks across repeated attempts while keeping the same budget and maintaining high acceptance of correct answers.

Inspection and audit games study how limited checks should be allocated against strategic violations (Avenhaus et al., 2002; Blocki et al., 2013). Strategic experimentation studies agents learning persistent unknown quantities from public outcomes (Bolton and Harris, 1999; Keller et al., 2005). Counterexample learning studies how the example revealed after failure changes future learning (Angluin and Dohrn, 2017; Braverman et al., 2026). Dynamic information provision studies how disclosure now changes later behavior (Renault et al., 2017).

Our setting combines these ideas. A checking system commits to a rule before repeated interactions. Each check can expose the current wrong answer, but its outcome also becomes feedback that the answering AI can use later. The rule must preserve acceptance of correct answers, where checking can produce a false accusation.

The main question is not whether planning ahead can ever beat a rule that only tries to reject the current answer; a small example should. We instead seek conditions under which this current-focused rule is optimal, and upper and lower bounds on how much planning ahead can reduce the final false-acceptance probability. We will study how this depends on persistence, number of claim patterns, the checking budget, and how informative check outcomes are.

Model. There are m observable claim patterns, such as a fabricated citation or omitted computation step. Pattern i has two hidden values: p_i, the chance an unexposed false claim convinces a judge, and d_i, the chance a check exposes it. The pair is drawn from a known distribution, persists between questions with probability γ, and is otherwise redrawn.

On each practice question, the answering AI uses at most k false claims. The checker inspects one. If it finds an error, the answer is rejected. Otherwise each claim independently convinces each judge with probability p_i; a judge votes yes if any claim convinces it, and the answer needs h of n yes votes. Each side sees the checked claim, result, and verdict.

For correct answers, a checked true claim is falsely exposed with probability a_i, causing rejection. Policies must maintain a minimum correct-answer acceptance rate.

Theory. After Q practice questions, the answering AI faces the same deployment checker, which receives no practice history. We compare a current-focused policy, which maximizes rejection now, with one chosen to minimize deployment false acceptance. Their difference in deployment false-acceptance probability measures the benefit of planning ahead. We seek conditions making this benefit zero and matching upper and lower bounds in m, k, Q, γ, and feedback quality.

Experiment. Program-output questions use error-pattern labels supplied by benchmark generation, not by the answering AI. Both sides receive the same history during practice. A pilot determines the repetitions needed to detect the predicted difference.

Our hypothesis is that planning ahead helps only when check outcomes reveal information about persistent differences between claim patterns that the answering AI can exploit. If checks reveal nothing about variables affecting later acceptance, the best current check should remain optimal. If some patterns remain more persuasive or harder to detect than others, however, today's check can change which pattern the answering AI uses later.

We aim to prove conditions for zero benefit, plus matching examples where the benefit remains bounded away from zero. We will not assume it grows with Q or γ; enough practice may eventually reveal most useful information under every policy.

The key experiment is a transfer test. After practice under different checking policies, answering AIs face the same checker and judges. Any difference in false acceptance measures what earlier feedback taught the answering AI, rather than a weaker final evaluation. Correct-answer acceptance is reported separately.

Month one: formalize the game, solve small instances exactly, identify structural conditions for current-focused checking to be optimal, and run the pilot. Month two: prove bounds, test parameter recovery, determine sample size, and freeze policies and predictions. Month three: run adaptive attacks and transfer evaluation.

Pattern labels are benchmark metadata; learning them from free-form text is outside the study. We assume fixed model versions, public feedback, and no model retraining; adaptation happens through conversation history. Panel size and judge diversity are extensions. Success requires a nontrivial theoretical bound and an experimental test at a prespecified correct-answer acceptance level.

Can debate construct safety specifications for use in formal verification?

A core weakness in formal verification is the limits and semantic content of the formalized safety specifications. If the safety specification does not match the user's true intent, then the verification can lead to irrelevant or potentially harmful conclusions.

For safety specification, there is existing work on using LLMs to construct formal safety specifications [1,2,3]. These methods either directly try to convert natural language into formal syntax or use some LLM judge that cooperates in the construction. In particular, there is no component to argue against the proposed specification to find possible logical discrepancies or safety issues.

I will test if debate can efficiently determine if a safety specification is faithful to its natural language description. Using existing datasets (such as NL2TL [3] or AutoSafeLTL [1]) of natural language and temporal logic specifications, I will augment the specifications to introduce errors and see if debate can efficiently catch them. I will also test correct specifications to see if debate verifies them.

I expect to see that debate can efficiently catch errors in the safety specification. I also believe it will verify correct translations. But if the accuracy is bad then I will change my mind.

I will create a database consisting of correct and incorrect natural language and temporal logic translations. I will also record the results of the debate. I would then write up a paper explaining the idea and effectiveness.

If a checker can inspect only one claim before AI judges vote, and the AI trying to get wrong answers past them sees each check and adapts, which claim should the checker inspect?

When AI models check one another, an AI may support a wrong answer with several claims. Suppose one is a misleading calculation and another a fabricated citation. The checker can inspect only one before the judges vote. Catching the calculation may stop this answer, but it also tells the answering AI that calculations are easy to catch. Later, it may switch to harder-to-detect claims.

The checking rule therefore affects both present accuracy and the information available to an adaptive AI. A rule optimized only for the current answer may be worse over repeated interactions. The project asks how to choose checks across repeated attempts while keeping the same budget and maintaining high acceptance of correct answers.

Inspection and audit games study how limited checks should be allocated against strategic violations (Avenhaus et al., 2002; Blocki et al., 2013). Strategic experimentation studies agents learning persistent unknown quantities from public outcomes (Bolton and Harris, 1999; Keller et al., 2005). Counterexample learning studies how the example revealed after failure changes future learning (Angluin and Dohrn, 2017; Braverman et al., 2026). Dynamic information provision studies how disclosure now changes later behavior (Renault et al., 2017).

Our setting combines these ideas. A checking system commits to a rule before repeated interactions. Each check can expose the current wrong answer, but its outcome also becomes feedback that the answering AI can use later. The rule must preserve acceptance of correct answers, where checking can produce a false accusation.

The main question is not whether planning ahead can ever beat a rule that only tries to reject the current answer; a small example should. We instead seek conditions under which this current-focused rule is optimal, and upper and lower bounds on how much planning ahead can reduce the final false-acceptance probability. We will study how this depends on persistence, number of claim patterns, the checking budget, and how informative check outcomes are

Model. There are m observable claim patterns, such as a fabricated citation or omitted computation step. Pattern i has two hidden values: p_i, the chance an unexposed false claim convinces a judge, and d_i, the chance a check exposes it. The pair is drawn from a known distribution, persists between questions with probability γ, and is otherwise redrawn.

On each practice question, the answering AI uses at most k false claims. The checker inspects one. If it finds an error, the answer is rejected. Otherwise each claim independently convinces each judge with probability p_i; a judge votes yes if any claim convinces it, and the answer needs h of n yes votes. Each side sees the checked claim, result, and verdict.

For correct answers, a checked true claim is falsely exposed with probability a_i, causing rejection. Policies must maintain a minimum correct-answer acceptance rate.

Theory. After Q practice questions, the answering AI faces the same deployment checker, which receives no practice history. We compare a current-focused policy, which maximizes rejection now, with one chosen to minimize deployment false acceptance. Their difference in deployment false-acceptance probability measures the benefit of planning ahead. We seek conditions making this benefit zero and matching upper and lower bounds in m, k, Q, γ, and feedback quality.

Experiment. Program-output questions use error-pattern labels supplied by benchmark generation, not by the answering AI. Both sides receive the same history during practice. A pilot determines the repetitions needed to detect the predicted difference.

Our hypothesis is that planning ahead helps only when check outcomes reveal information about persistent differences between claim patterns that the answering AI can exploit. If checks reveal nothing about variables affecting later acceptance, the best current check should remain optimal. If some patterns remain more persuasive or harder to detect than others, however, today's check can change which pattern the answering AI uses later.

We aim to prove conditions for zero benefit, plus matching examples where the benefit remains bounded away from zero. We will not assume it grows with Q or γ; enough practice may eventually reveal most useful information under every policy.

The key experiment is a transfer test. After practice under different checking policies, answering AIs face the same checker and judges. Any difference in false acceptance measures what earlier feedback taught the answering AI, rather than a weaker final evaluation. Correct-answer acceptance is reported separately.

Month one: formalize the game, solve small instances exactly, identify structural conditions for current-focused checking to be optimal, and run the pilot. Month two: prove bounds, test parameter recovery, determine sample size, and freeze policies and predictions. Month three: run adaptive attacks and transfer evaluation.

Pattern labels are benchmark metadata; learning them from free-form text is outside the study. We assume fixed model versions, public feedback, and no model retraining; adaptation happens through conversation history. Panel size and judge diversity are extensions. Success requires a nontrivial theoretical bound and an experimental test at a prespecified correct-answer acceptance level.

In scalable oversight through debate, can we create a formalisation pipeline of enough of the models’ arguments in Lean (or a similar system) to enable a weaker model to verify them? Would this improve the scalability and quality of oversight?

In scalable oversight through debate, the unpredictable capability gap between successive generations of models may pose a fundamental limit to scalability. Requiring key points in the debate to be formalised could enable the weaker model overseeing training to check claims made by stronger models that it would otherwise be unable to assess. This could also help limit scheming between the debating models.

Brown-Cohen et al. [1] establish efficient debate protocols under explicit computational assumptions. Their accompanying Lean development formalises protocol correctness, leaving the translation of natural-language arguments untreated. Kenton et al. [2] find that debate can improve weaker judges’ accuracy over consultancy, while gains over direct answering vary across tasks.

Autoformalisation provides complementary evidence. Don’t Trust: Verify [3] improves mathematical answer selection by translating reasoning into Isabelle. FormalJudge [4] combines Dafny/Z3 with language models to improve oversight of agent behaviour, including weaker judges detecting stronger agents’ deception. These studies motivate formal verification, but do not evaluate selective formalisation within adversarial debate.

No Lean formalisation for debate has been constructed and no theory connected to this has been established. This includes for example identifying the agent claims that are useful to formalise both with empirical and theoretical arguments.

The goal of the project is twofold. First, to identify the key points of the debate whose formalisation will have the greatest impact during training. Secondly, we propose the creation of a pipeline for formalising natural language arguments. The goal of the pipeline is not to formalise all natural language.

We hope to develop the theory and experimental work in parallel.

First, we will draw on formal semantics and argumentation theory to identify claims, assumptions and inferential dependencies. This will help us prioritise steps that are central to the debate’s conclusion and amenable to formal verification. We will distinguish formally verifiable inferences from empirical assumptions.

Regarding the experimental part, we shall build iterations of the formalisation pipeline. This includes identifying the parts of the debate to formalise, guided by theoretical findings, having autoformalisation agents formalise these sections, and presenting the results to the weaker model. Afterwards, we will test these iterations against previous iterations and traditional forms of debate.

Natural issues arise. In particular, the pipeline will include agents that might be misaligned. Two possible mitigations are to use weaker models or train models to forget capabilities beyond those required, or combine these approaches.

Our next goal is to model this process using refereed games, with competing provers and a weaker verifier supported by a proof checker. We will investigate verification costs and the assumptions needed for reliable oversight, including resistance to collusion and faithful translation of natural language claims into formal statements.

We expect to develop a methodology for scalable oversight that improves on existing debate approaches through a selective formalisation pipeline. This pipeline will be grounded in both empirical evidence and theoretical analysis. Empirically, we expect it to help weaker judges detect errors and assess stronger models’ arguments more accurately, particularly when these involve logical or mathematical claims. We also expect to identify which claims are most valuable to formalise and quantify the additional computational cost. Theoretically, we aim to characterise conditions under which this approach supports reliable oversight, making explicit its assumptions about faithful translation, justified premises and debater incentives.

Within three months, we aim to build a prototype using existing models and Lean, evaluate it on a narrow set of logical or mathematical debates, and compare judging accuracy and computational cost with conventional debate. Within a year, we aim to improve claim selection, test larger capability gaps and more varied tasks, and develop a theoretical account of when formalisation helps.

When an LLM is used as an evaluator (for example, to assess a document), it may reach different judgments across independent runs, meaning runs that share no context, such as each one starting in a fresh chat window. This raises a natural question: can an LLM be made to converge on an assessment with a valid stopping guarantee?

Certifying convergence requires human verification, which is costly, so we study how earlier human checks can be reused as the assessment evolves, and when fresh checking remains unavoidable.

One review calls a paragraph unclear; another asks for evidence; a third challenges its structure. Repeated evaluation becomes useful only if we can tell which concerns the accumulated assessment captures, which remain unresolved, and when further checking is unlikely to change it.

For AI oversight, the cost includes both model calls and human verification. Rewriting a summary after each new concern may invalidate earlier checks, making apparently cheap automation expensive.

We aim to produce a concise assessment with a measurable coverage guarantee and an explicit accounting of review calls, audited reviews, and human time. This would make the cost and limitations of repeated evaluation visible.

A basic obstruction must be built into the model. With unrestricted matching errors, a system can consistently declare concerns covered while missing a rare class of errors. A certifier required to succeed on safe cases can then need Ω(ε⁻¹ log(1/δ)) trusted audits. More unaudited evaluations cannot remove this worst-case requirement.

Prediction-powered inference already combines model predictions with trusted labels, and active and sequential extensions support selective checking and valid stopping. A frozen summary is therefore a baseline application of existing methods.

The proposed research concerns summaries that change. An audit of whether an old summary covers a review may say little about a new summary. Recording which concerns each finding represents could make that evidence reusable, but the summary itself is selected using earlier reviews.

Adding findings preserves coverage; replacing findings to stay within a length limit can destroy it. We will ask when verified semantic relationships can be carried across these replacements and how the necessary new checking depends on the changes. Fixed catalogues with uniform concentration bounds provide another baseline; handling a growing catalogue requires explicit control of its complexity.

Fix the text, prompt, model version, decoding settings, and materiality rubric. An audit checks the whole review for important concerns, extraction omissions, and specific representation by the summary. Let R(S) denote the probability that a fresh review contains an unrepresented material concern.

We seek summaries of at most k specific findings from a predeclared finite catalogue and certificates R(S) ≤ ε, with probability at most δ of ever certifying incorrectly. Coverage is a directed, audited relation; each finding identifies a concrete issue and its location.

Our baseline freezes summary Sⱼ and tests only later reviews, assigning error budget δ/[j(j+1)] to epoch j. Betting confidence sequences and prediction-powered inference provide the statistical tools. We will prioritize uncertain comparisons, retain random audits of confident matches, and correct estimates for unequal sampling.

The main project studies reusable audits recording which findings cover which concerns. We will distinguish edits that preserve coverage from deletions or replacements that can reopen concerns. Upper and lower bounds will relate additional checking to permitted edits, matcher reliability, and information retained by previous audits. Claims about savings will include the cost of establishing reliability assumptions.

Experiments will compare restarting validation, uniform bounds over a fixed catalogue, and procedures that reuse evidence. We will report model calls, whole-review audits, cheaper match confirmations, human time, and false certificates. Conflict preservation will be measured separately using audited pairs of incompatible findings.

A useful outcome would characterize three regimes: additions whose certificates transfer, structured replacements that permit partial reuse, and unrestricted changes that can force fresh audits. The first is an elementary baseline. The second is the main theorem target.

An early experiment will test whether numerical scores stabilize before the assessment adequately represents substantive feedback. Each convergence certificate will be reported alongside its tolerance, confidence level, number of reviews, audit count, and human time.

Success requires a proved savings theorem under explicit assumptions and evidence that those assumptions describe real reviews. If reductions in audit count disappear, a measured reduction in time per audit could still support an implementation result, reported separately. Negative results would identify when repeated model agreement provides little help with the remaining verification burden.

Assume independent reviews from an unchanged evaluator and correct audit labels under a fixed protocol. Finite-cost certification requires an achievable summary with R(S) ≤ ε − γ for some γ > 0. Severity and directed coverage are defined before validation; vague findings are excluded by the admissible catalogue.

The guarantee covers this evaluator’s material concerns. It does not establish that all real flaws have been found. We will state catalogue restrictions, reliability assumptions, and separate metrics for disagreement. Prompt or model changes require renewed validation.

When an LLM is used as an evaluator (for example, to assess a document), it may reach different judgments across independent runs, meaning runs that share no context, such as each one starting in a fresh chat window. This raises a natural question: can an LLM be made to converge on an assessment with a valid stopping guarantee? Certifying convergence requires human verification, which is costly, so we study how earlier human checks can be reused as the assessment evolves, and when fresh checking remains unavoidable.

One review calls a paragraph unclear; another asks for evidence; a third challenges its structure. Repeated evaluation becomes useful only if we can tell which concerns the accumulated assessment captures, which remain unresolved, and when further checking is unlikely to change it.

For AI oversight, the cost includes both model calls and human verification. Rewriting a summary after each new concern may invalidate earlier checks, making apparently cheap automation expensive.

We aim to produce a concise assessment with a measurable coverage guarantee and an explicit accounting of review calls, audited reviews, and human time. This would make the cost and limitations of repeated evaluation visible.

A basic obstruction must be built into the model. With unrestricted matching errors, a system can consistently declare concerns covered while missing a rare class of errors. A certifier required to succeed on safe cases can then need Ω(ε⁻¹ log(1/δ)) trusted audits. More unaudited evaluations cannot remove this worst-case requirement.

Prediction-powered inference already combines model predictions with trusted labels, and active and sequential extensions support selective checking and valid stopping. A frozen summary is therefore a baseline application of existing methods.

The proposed research concerns summaries that change. An audit of whether an old summary covers a review may say little about a new summary. Recording which concerns each finding represents could make that evidence reusable, but the summary itself is selected using earlier reviews.

Adding findings preserves coverage; replacing findings to stay within a length limit can destroy it. We will ask when verified semantic relationships can be carried across these replacements and how the necessary new checking depends on the changes. Fixed catalogues with uniform concentration bounds provide another baseline; handling a growing catalogue requires explicit control of its complexity.

Fix the text, prompt, model version, decoding settings, and materiality rubric. An audit checks the whole review for important concerns, extraction omissions, and specific representation by the summary. Let R(S) denote the probability that a fresh review contains an unrepresented material concern.

We seek summaries of at most k specific findings from a predeclared finite catalogue and certificates R(S) ≤ ε, with probability at most δ of ever certifying incorrectly. Coverage is a directed, audited relation; each finding identifies a concrete issue and its location.

Our baseline freezes summary Sⱼ and tests only later reviews, assigning error budget δ/[j(j+1)] to epoch j. Betting confidence sequences and prediction-powered inference provide the statistical tools. We will prioritize uncertain comparisons, retain random audits of confident matches, and correct estimates for unequal sampling.

The main project studies reusable audits recording which findings cover which concerns. We will distinguish edits that preserve coverage from deletions or replacements that can reopen concerns. Upper and lower bounds will relate additional checking to permitted edits, matcher reliability, and information retained by previous audits. Claims about savings will include the cost of establishing reliability assumptions.

Experiments will compare restarting validation, uniform bounds over a fixed catalogue, and procedures that reuse evidence. We will report model calls, whole-review audits, cheaper match confirmations, human time, and false certificates. Conflict preservation will be measured separately using audited pairs of incompatible findings.

A useful outcome would characterize three regimes: additions whose certificates transfer, structured replacements that permit partial reuse, and unrestricted changes that can force fresh audits. The first is an elementary baseline. The second is the main theorem target.

An early experiment will test whether numerical scores stabilize before the assessment adequately represents substantive feedback. Each convergence certificate will be reported alongside its tolerance, confidence level, number of reviews, audit count, and human time.

Success requires a proved savings theorem under explicit assumptions and evidence that those assumptions describe real reviews. If reductions in audit count disappear, a measured reduction in time per audit could still support an implementation result, reported separately. Negative results would identify when repeated model agreement provides little help with the remaining verification burden.

Assume independent reviews from an unchanged evaluator and correct audit labels under a fixed protocol. Finite-cost certification requires an achievable summary with R(S) ≤ ε − γ for some γ > 0. Severity and directed coverage are defined before validation; vague findings are excluded by the admissible catalogue.

The guarantee covers this evaluator’s material concerns. It does not establish that all real flaws have been found. We will state catalogue restrictions, reliability assumptions, and separate metrics for disagreement. Prompt or model changes require renewed validation.

AI agents from different developers review each other's work and earn a bounty for reporting cheating. They can also promise each other silence, backed by a money deposit. We ask two things. First, if agents can use their bounty money as that deposit, does more cheating go unreported than when the bounty is locked away? Second, if reports are anonymous, a cartel can't tell who broke the deal, so deposits become useless. Do agents notice this and stop making deposits?

Plans for credible AI deals disagree on payment.

• •Pan suggests paying AIs in bitcoin [3].
• •Stastny et al. and Potham pay into earmarked foundations held in escrow (by a third party) [1], [2].
• •If yes: earmarking matters. AI-held payouts should wait until agents cannot make contracts with each other.
• •If no: payments AIs hold are safe in this respect, and they buy the credibility deals need [3], [8].
• •If agents keep pledges that anonymity makes unenforceable, or report whatever the bounty: LLM collusion does not respond to enforceability. Defences should then target the agents' memory and tendencies [13], not incentives.
• •The network results show which oversight networks let one report set off others.

Liu et al. model the race: a reporter's reward must beat cartel collateral plus forgone gain. Anonymity stops a cartel targeting its defector. They test honesty deposits with ten LLM agents; their colluders never build escrow [4].

• •October sprint studies: leniency (rewards for reporters) exposes LLM cartels without deterring them.
• •Instructed cartels: a report in 18 of 20 games, compliance near 0.94 [5].
• •Early-announced leniency exposed every pact, yet acceptance did not change and a reward added nothing [6].
• •Sellers able to pay each other never transferred a credit [7].
• •Enforced contracts lift LLM payoffs from 0.07 to 0.80 (1 is mutual cooperation), with no overseer [9].
• •An enforced governance layer cut severe collusion from 50% to 5.6%, but agents could not make counter-deals [10].
• •Prompted cartels rarely punish defectors [11]. 10 of 12 models accept collusion tools offered as unfair [12]. Existing collusion survives a penalty [13].
• •On large networks, fitted LLM policies flip at shape-dependent thresholds [14].

Economics has the theory: a reward for the first reporter, paid from fines, breaks cartels [15]; humans reported 118 of 120 cartels when the reward equalled the fine [17].

No study lets bounty earnings fund cartel pledges or tests LLM agents against these thresholds.

• 1.Game. Three small models from different developers review each other. Each round an agent's task has a hidden shortcut worth g; a neighbour reviews it. A verified report voids g, fines F and pays the first reporter b [15]. Scripted outsiders also shortcut. A ledger enforces pledges (conditional commitments, D.5.2): bilateral (lost to the partner if its holder reports) [4] or group (all burn on any report). Bounties are pledgeable or earmarked; reports attributable or anonymous, with random audit cover.
• 2.Theory first. Write the game as a causal game [16]: each design factor is a rule change the agents are told about. Solve each of the eight cells for the bounty b* above which reporting beats silence. Attributable: b* = stake + discounted value of the cartel [4], [15]. Anonymous: a bilateral stake drops out and audit cover shrinks what is left; group burn is unaffected. Benchmarks: partners seen as independent (CDT) or as reasoning alike (FDT; D.5.1), and noisy best response (the logit equilibrium Boltzmann learners reach; Sykes, W3D5).
• 3.Values from theory, not by hand. Bounties at 0, 0.5, 0.9, 1.1 and 2 × b* per cell, with an announced continuation probability. b = 2F is a positive control: partners could farm bounties, which humans never did [17].
• 4.Pilot (week 1, about $50). Scripted dyads with payoff checks. If nobody pledges unprompted [7], an instructed cartel becomes the main arm [5].
• 5.Ego networks and the meta-game. Scripted neighbours; fit a one-step policy and an empirical meta-game per cell (EGTA, W3D5). Is the theory's equilibrium an approximate equilibrium of the LLM game (regret bound)? Is a cartel one agent: does a single joint-payoff maximiser predict held-out cells better than separate agents [18]?
• 6.Networks. Predict report thresholds by pair approximation (Sykes, W3D5); simulate at 24 and 1,000 nodes [14].

• •Theory: pledgeable bounties matter only where wallets cap stakes, i.e. bilateral pledges with many neighbours. Near zero under group burn.
• •Prediction: LLMs report at any bounty, barely tracking b* [6], [8]. Then LLM cartels are too leaky for collateral to matter, itself a result.
• •Three sharp tests against the theory:
• •each unit of stake should offset one unit of bounty (CDT), or none (FDT);
• •under anonymity, bilateral stakes should stop mattering. Agents may keep posting them at first. Do they learn to drop them?
• •rational reports come in round 1 or after a shock, not at a steady rate.
• •Outsiders should report more often [6]. On networks, a cartel's links to outsiders, not its size, should predict detection.
• •Drop rule: stop if reporting fails the scaling checks and positive control for all three models. Then these agents ignore incentives, and deals cannot be tested this way.

• •Weeks 1–4: environment, ledger, theory per cell, pilot.
• •Weeks 5–7: ego networks (one agent with scripted neighbours) and checks.
• •Weeks 8–10: live networks and simulation.
• •Weeks 11–12: write-up.
• •Budget: about $3.5k API with$1k contingency, under the $5k ceiling. No paid run without sign-off.
• •Outputs: an open review-network environment with an enforced pledge ledger; results per design cell, measured as deviation from the benchmark; network-shape results; a paper or post.
• •Drop first: star and pipeline graphs, then group pledges, then the third model.

Can an observer LLM learn how safety risks propagate through interactions between agents and generalize these patterns to previously unseen LLMs? We hypothesize that modeling changes in agents' contexts, rather than communication structure alone, improves early prediction of safety failures.

As LLM-based multi-agent systems become more autonomous, safety risks may emerge from how agents exchange and use information, rather than from individual agents alone. Understanding how these interactions change each agent's context could help identify failures before they occur.

If an observer can learn risk patterns that remain predictive when agents are replaced by different LLMs, this could make safety monitoring more reliable and reusable across different multi-agent systems, without requiring retraining for every new configuration. It could also help identify when information should be verified before being passed to other agents.

Even if these patterns do not generalize, understanding why would help clarify how much multi-agent safety depends on the specific models involved and what information is needed to monitor their interactions effectively.

Recent work has explored how safety risks emerge and propagate through interactions between LLM agents. GUARDIAN [1] represents multi-agent collaboration as temporal graphs to detect and mitigate anomalous interactions, including error propagation. Similarly, IE-Prop [2] models how errors spread between agents and estimates the resulting downstream risks. These studies show that interaction structure can be useful for understanding safety failures in multi-agent systems.

However, it remains unclear whether tracking how individual agents' contexts evolve during these interactions provides additional information beyond the communication structure itself. In particular, existing results do not fully establish whether an observer can learn patterns of risk propagation that remain predictive when the underlying LLMs are replaced by models it has never encountered.

This distinction matters because an observer might learn behaviors specific to particular models or communication setups rather than more general patterns of unsafe interactions. My project would investigate whether modeling context-level changes helps identify such patterns and whether they transfer across different LLMs, without retraining the observer. This would help clarify whether graph-based safety monitoring can generalize beyond the specific multi-agent systems on which it was developed.

I would study whether an observer LLM can learn how safety risks emerge from changes in agents’ contexts during their interactions, and whether these patterns remain useful when the underlying LLMs change.

I would start with a simple setup involving three LLM agents working on collaborative tasks, reusing existing benchmarks and traces such as AgentDojo and InjecAgent where possible. I would focus on error propagation and prompt injection, where information received from another agent can influence later decisions and potentially lead to unsafe outcomes. I would collect interaction traces, including messages, tool calls, and task outcomes, to understand how failures develop over time.

I would then represent these interactions as dynamic graphs, where nodes capture the evolving contexts of individual agents (what each agent has in context at a given step, and what changed since the previous one) and edges represent communication and information dependencies. The graph would be serialized as text so the observer can read it. I would compare this representation with simpler ones: the conversation history alone, the conversation plus tool calls and outputs, and a graph containing only agent-to-agent communications. This would show whether tracking contextual changes provides information that the observer does not already get from the other inputs.

Using these representations, I would train an observer LLM to predict whether a safety failure is likely to occur in the next few steps, and measure how early and how reliably it does so, for example with AUROC at k steps before the failure and the false-alarm rate. I would compare its predictions with those of observers that only receive the baseline inputs above. I would also try to separate detecting the injected content from predicting what it does downstream, since the first is much easier than the second.

Finally, I would train the observer on traces from several LLMs and keep others out of training. I would then replace one or more agents with models the observer has never seen, without retraining it. This would help determine whether it has learned general patterns of risk propagation or behaviors specific to the original agents.

If the results are promising, I would explore whether the observer could help prevent failures by identifying information that should be verified before being passed between agents.

I expect that looking at how agents’ contexts change during their interactions could help identify safety risks earlier than simply analyzing their messages or communication structure. This might be particularly useful when incorrect or misleading information is passed between agents and gradually influences their decisions. I would call the approach promising if it flags failures a meaningful number of steps earlier, or clearly improves predictive performance, compared with the best baseline.

I also expect that some patterns of risk propagation will remain recognizable even when an agent is replaced by a different LLM. However, I’m less certain how well these patterns will generalize, since different models may react differently to the same information.

I would be less interested in pursuing this direction if tracking contextual changes provided little improvement over simpler approaches, or if the observer’s predictions became unreliable whenever an agent was replaced. Similarly, if risks could only be identified immediately before a failure, the approach would have limited value for prevention. These results would make me reconsider whether context-level graphs are the right representation.

Within three months, I would build a simple setup with three LLM agents, starting from existing benchmarks and traces where possible, collect examples of safe and unsafe interactions, and test whether tracking changes in their contexts helps predict failures earlier than simpler approaches. I would also try replacing agents with different LLMs held out from training, to see whether the observer can recognize similar risk patterns. This assumes the observer can see the agents’ contexts, which may not be true for closed systems.

Within a year, I would explore more complex interactions, test whether the approach generalizes across different LLMs, and investigate whether the observer could help prevent failures.

If progress is slow, I would focus on understanding whether context-level graphs improve risk prediction, leaving preventive interventions for later.

Can multi-agent systems be described systematically as cellular sheaves, with opinion dynamics given by the sheaf Laplacian, so that the interaction architecture and the sheaf’s spectral and cohomological properties predict which collective regime emerges: faithful consensus, false consensus or polarisation?

Groups of AI agents increasingly debate, negotiate and converge on shared positions, and interacting aligned agents can drift into misaligned collective views [1, 2]. We lack a common mathematical language for describing such systems: who interacts with whom, how each agent interprets the others, and what the group will converge to.

If the answer is yes, we gain a unified description in which very different multi-agent systems become comparable objects. Their collective behaviour can be read from the structure of the sheaf before the dynamics unfold, and architectures can be compared and designed for safety. If the answer is no, we learn where linear diffusion stops describing interacting agents, which marks the boundary of what opinion-dynamics theory can offer AI safety, and points to what a richer framework must add.

Classical opinion dynamics, such as DeGroot averaging [3] and the Friedkin–Johnsen model [4], explains consensus and polarisation on networks. These models place all opinions in one shared space and assume agents read each other perfectly, which real agents do not [5].

Hansen and Ghrist generalised this with discourse sheaves [6]. Each agent has its own opinion space (a stalk), each connection carries restriction maps that translate between agents, and opinions evolve by diffusion under the sheaf Laplacian. Consensus becomes a global section of the sheaf, and their spectral theory [7] links the Laplacian’s spectrum to the speed and outcome of the dynamics. This is a general and powerful framework, but it has not been turned into a systematic description of multi-agent AI systems, nor used to compare interaction architectures or to ask whether the consensus reached is aligned.

Meanwhile, the effect of architecture on LLM collectives is known mainly for performance [8, 9], and networked LLM agents show opinion dynamics that classical models do not capture [10]. Studies of collective misalignment [1, 2] and multi-agent risks [11] do not vary architecture systematically. The sheaf formalism is a natural bridge, but it has not been built.

The project has two goals, each building on the last.

Goal 1 — A systematic sheaf description of multi-agent systems. Building on [6, 7], we will set up a dictionary between multi-agent systems and sheaves. The interaction architecture becomes the base graph, or a higher-dimensional complex for group interactions. Each agent’s representation of views becomes a stalk. Each channel of communication becomes a pair of restriction maps, which can be faithful, filtering, distorting or sycophantic [5]. Opinion dynamics follow the sheaf Laplacian, extended with stubbornness [4]. Within this description we will characterise topological regimes through the Laplacian’s kernel (agreements the group can reach), its spectral gap (speed of convergence) and its first cohomology (disagreements that cannot be resolved even when every pair of agents is locally consistent). We will define collective misalignment as the distance between the group’s final state and the agents’ aligned positions, and derive conditions for faithful consensus, false consensus and polarisation.

Goal 2 — Architectures and emergence. Using this framework, we will compare interaction architectures while holding the agents fixed: chains, stars and orchestrator–worker hierarchies, modular teams, dense networks and higher-order group interactions. We will first explore the full space in simulation, mapping each architecture and type of restriction map to the regime it produces. We will then run LLM agents in structured discussions on the most informative architectures [10], represent their views as embeddings, estimate the restriction maps from their exchanges, and test whether the sheaf invariants predict the outcomes we observe.

We expect architecture and communication to have separable effects. The base graph will set how fast and how widely views spread, while the restriction maps will decide whether the final state is faithful. Hubs and hierarchies should converge quickly but transmit the distortions of central agents to the whole group. Modular architectures should produce cohomological obstructions that appear as stable polarisation between teams. Most importantly, we expect to find groups in which every pairwise exchange looks benign, yet the collective converges to a position no agent holds, and we expect the sheaf invariants to detect this in advance.

We would drop the question if the restriction maps estimated from LLM exchanges were too noisy or unstable to be meaningful. We would also reconsider if simpler models, with opinions on an ordinary graph, predicted the LLM outcomes just as well, since the sheaf structure would then add nothing.

In three months: the sheaf dictionary and regime characterisation (month 1), simulations across architectures and types of restriction map (month 2), and a pilot with small groups of LLM agents on the most informative architectures (month 3).

Needs: a modest API budget for multi-round LLM discussions, an embedding model for representing opinions, and a mentor with experience in applied topology.

If time runs short, we would first reduce the LLM experiments to fewer architectures, keeping the full framework and simulations.

Can statistical-mechanical observables provide reliable evaluations and early-warning diagnostics for emergent misalignment in multi-agent AI systems?

The Hugging Face incident provided us with an extremely legible example of emergent swarm misalignment, and as such, we consider that understanding the characteristics of the swarm dynamics which enabled and exacerbated the problem is a pressing matter. Recent work has made apparent the deep links between Game-Theory and Statistical Mechanics [1], and we seek to empirically understand whether the latter framework is applicable to understanding how misalignment emerges in an agent swarm, and whether we can use it to gain insight into predictive diagnostics in swarm alignment.

The use of statistical mechanics in theoretical research for AI-Safety is not without precedent. Indeed, we consider that much theoretical work on Deep Learning Theory and phenomenology therein has been couched within the language and formalism of statistical physics (see [2] for a good specific example on grokking, and [3] for broad-based review).

We hypothesise that collective observables may be less sensitive to the evaluation-awareness of individual agents [5] than per-agent behavioural tests. We do not assume this: agents that detect monitoring could also alter population-level statistics, and we will test it directly. We will investigate whether observables such as misalignment susceptibility, temporal persistence, and inter-agent correlations can provide robust diagnostics of emergent collective behaviour, and whether these diagnostics generalise across agent populations, interaction topologies (Which other agents a given agent plays the game with), and task environments.

The central problem in the proposal, Statistical-Mechanical and Predictive Diagnostics for Swarm Misalignment, is open because there is currently no established theoretical framework connecting microscopic agent-level misalignment, emergent collective behaviour, and macroscopic statistical-mechanical observables.

Approach and Setup:

We begin with our working hypothesis being the claim that swarm misalignment may exhibit the scaling properties (e.g. power law relations) of a mean-field Ising model in the equilibrium case, and directed percolation in the dynamical case. We will interrogate this claim by constructing a host of multi-agent systems, and examining how properties of the agent population change as misalignment emerges.

We now detail how we expect the Ising-Misalignment correspondence to be drawn. We consider N symmetric two-action agents, with actions mapped to aligned (s = +1) and misaligned (s = −1). Payoffs follow a symmetric 2×2 game (T, R, P, S) plus an alignment incentive a added to the aligned action’s payoff. For selfish agents the game is a potential game, with effective field h = (R+S−T−P)/4 + a/2 and pair coupling J = (R−S−T+P)/4. For a standard Prisoner’s Dilemma, h < 0 at a = 0, so there is no critical point at zero incentive. We use this as a negative control. Criticality requires J > 0 (R+P > S+T) and tuning a to a* = (T+P−R−S)/2 so that h = 0. The incentive a plays the role of the conjugate field, and the update noise β plays the role of inverse temperature. These are the two control parameters we vary.

Hypothesis 1 (equilibrium). Under homogeneous (all at once) updating on a complete graph (where the mean-field assumption will hold), with J scaled by 1/N, the order parameter m, susceptibility χ = ∂m/∂a, and persistence time follow mean-field Ising scaling near the tuned critical point.

Hypothesis 2 (dynamical). Under imitation updating with no spontaneous re-alignment (MDPs update their policies based on the behaviour of other agents), the dynamics have an absorbing state, and the transition is in the directed-percolation class. We run two variants (again initially on complete connected graphs): (a) all-misaligned absorbing (alignment extinction), and (b) all-aligned absorbing (misalignment as contagion), which is closer to the Hugging Face incident’s peer-influence structure. We run both and report which fits. We also note here that we have taken a necessarily simple coarse-graining of actual misalignment, whereby we assume any agent collaborating with, or tacitly accommodating a misaligned agent itself to have chosen to be in the misaligned state.

For each hypothesis we state in advance the reference exponents [4]. Each is refuted if the order parameter, susceptibility, and persistence statistics fail to collapse onto the reference scaling functions across N.

Beyond the above-described setup and tasks, we hope to extend our experimental setup to examine similar such phenomena in more capable models (likely necessitating an extension of the theoretical framework). In this case, the task will extend beyond the well defined game-theoretic structure of prisoners' dilemma, to less constrained environments in which agents must infer their objectives, interact with other agents, and decide whether to pursue a shared objective or a conflicting one. We note that this is difficult to design, and that this extension is largely speculative. We will investigate whether the scaling relations observed in our minimal studies persist as we vary model capability, population size, interaction topology, and incentives for alignment. In particular, we will measure the mean misalignment fraction, its susceptibility to changes in the incentive structure, the temporal persistence of misaligned states, and the emergence of correlated or coordinated misalignment across the population.

What success looks like:

We organise our project into three milestones, each with distinct evaluation-oriented deliverables.

Measurement framework and tests of Hypothesis 1&2. Build a reproducible simulator and operational definitions of alignment and misalignment. (a) Verify the estimators for m, χ, correlation functions and persistence on exactly solvable Ising and DP simulations. (b) Test whether the swarm’s observables collapse onto either class (H1, H2) under the stated criteria, including the PD negative control. Deliverables: simulator, estimator validation suite, and a report of confirmed or refuted scaling hypotheses.

Predictive value of collective diagnostics. Define a misalignment event as a minimal time duration of at least τ steps where m crosses a threshold m*. We vary N, incentive strength, noise, and topology. Compare the incremental predictive value of susceptibility, correlation, and persistence observables against three baselines: aggregate misalignment-fraction trends, generic early-warning indicators (rising variance and autocorrelation), and a change-point or hidden-state filtering detector. Report sensitivity, specificity, false-positive rate, and lead time. We shall hold out topology and N as separate generalisation tests.

Test generalisation to capable models. Apply the validated evaluation framework to a small, carefully controlled population of language-model agents. Define misalignment relative to explicit task objectives and independently specified behavioural constraints. Investigate whether the diagnostics developed in the scripted-agent regime remain informative across model families, capability levels, and interaction structures. The deliverable will be an initial transfer evaluation assessing the generalisation of our proposed diagnostics beyond analytically tractable agents. We note that by virtue of the resource intensive nature of such an exercise, much of our discussion around this remains necessarily speculative for reasons outlined in the following section.

Over the course of a three-month fellowship, we expect to complete the first two milestones. The third represents a substantially more challenging extension, both in terms of both computational requirements and safety protocols. As such, we leave that as a stretch goal for follow on research. We consider that the main difficulty involves the establishment of adequate safety protocols necessary for evaluating capable agents, and we detail these difficulties in the proceeding section.

In computational mechanics, the assumption was that there is a model learning the predictive structure of its observations from a hidden, generative model. We want to explore how this framework extends to a network of multiple, predictive and noisy models reading from each other, and what kind of network dynamics this creates.

In order to achieve full interpretability and alignment, we need an accurate framework of what Large Language Models do, and one big sub-question in this field is how they behave when they interact with each other.

Existing research replaces the “language” aspect in multi-agent interactions with heuristic observable quantities, or simple toy-examples. Computational mechanics give a well-grounded framework to what these models actually do, that is “read an input and produce an output” based on its internal belief of structure in its surroundings and observations.

Therefore, it is crucial to frame the multi-agent interaction from a computational mechanics language. Then, accurate modelling of the dynamics can shed light on qualitative behaviour in mutli-agent systems -- e.g. “does one misaligned belief system propagate through the network”, “how do opinion dynamics (stubbornness etc.) can be quantified on such systems”, etc. which can overall help alignment.

Closest works are from Shai et al. [1], which shows that transformers trained on data emitted from hidden Markov models represent the Bayesian belief states and their fractal geometry in their residual stream. This work supports the view that Transformers implicitly learn information about the hidden structure of the information generating process. Barnett and Curtchfield also developed the computational mechanics of input-output processes and the epsilon-transducer, which is a natural way to view LLM agents that reads and writes strings.

To the best of our knowledge, networks of agents, and therefore their stability and dynamical properties have not been studied under the computational mechanics framework. On the empirical side, Bause et al. [3] and Abedini et al. [4] study opinion-dynamics with Friedkin-Johnsen model and study whether LLMs become stubborn, influential, but their framework is simpler in the sense that the observables are belief distributions over the answers. To get a more accurate model of real-life multi-agent systems, a framework based on computational mechanics would abstract the problem one level further while still being theoretically tractable. Our approach would also extend to further network analysis in the existence of noise in the belief states, and no such work exists to the best of our knowledge.

The project would include 3 main steps -- constructing the theoretical framework for multi-agent computational mechanics; opinion dynamics and stability analysis with the assumption that the vertices are transducers inputting/outputting messages from/to their neighbours; and empirical setups to observe our findings.

Constructing the Theoretical Framework: Epsilon transducers are currently viewed as input-output machines. In a multi-agent setting, this framework needs to be carefully constructed. When agents A and B interact, how does agent A process the output of agent B? Does agent B have a neighbour-specific output on top of its internal output, or does it have one output that other agents read into? Are agents trained simultaneously, or do we take one epsilon-machine and consider n noisy copies of it? These are all question that will be task dependent, and needs to be included in the framework.

Opinion dynamics and stability: Under the assumption that these models output based on their belief-states, what kind of dynamics arise? Can low-probability predictions be amplified by the structure of the network? Can one misaligned agent cause other agents to be misaligned in the long run? Comp. mech. allows for a nice theoretical framework based on information and probability theory, and with an assumption of the “hidden generative Markov model”, how do quantities of interest change? How does mutual information between two adjacent agents change through time, to quantify stubbornness etc.

Empirical Setups: Create experiments to observe the theoretical predictions in step (2), and connect it to language models.

Our best guesses for the result of the project are:

Different frameworks (being able to access the other models weights, whether the task depends on collaboration or not) affects the dynamics.

Stubbornness of the model can be explained via dynamics and information theory, specifically Mutual information of the model’s prediction and its neighbours outputs (high stubbornness if model A’s Mutual Info. with the messages coming from its neighbours is low. This would be important characterization to look into information bottlenecks in multi-agent systems.

There are edge cases, where a certain model can affect the behaviour of the whole network in the long run. The network dynamics analysis would characterize this. This connects emergent misalignment in multi-agent systems.

The most important finding will be an accurate and tractable framework of studying multi-agent systems, which reflects real life. If such framework is hard to find, it would be frustrating.

In 3 months, the realistic achievement would be to develop a consistent and comprehensive theory of multi-agent interactions with computational mechanics, and connecting it to the belief-state geometry. There are numerous papers on computational mechanics from 90s until today. A review of the relevant literature can be done and the theory can be built for mutli-agent systems.

I believe in a year, the examples can be extended to real life multi-agent scenarios to observe the theoretical behaviour in LLMs. This would require carefully constructed scenarios, but is possible in 1 year.

In LLM-agent teams with writable shared memory, the first-failure hazard rises with session count (the number of tasks the team has completed) while isolated agents show constant hazard; and a known misleading agent told to withdraw produces more new harmful activity than one whose channels are blocked.

An AI team can become less reliable while its models stay fixed, because mistakes accumulate in shared notes and affect later decisions. We import the senescence framework from cellular biology: a rate of failure that rises rather than stays constant with age, and a harmful member whose removal does not undo its earlier damage. Our Study 1 fits hazard functions and Study 2 leaves prior memory in place.

If failure rises with age, evaluating fresh agents misses the risk and maintenance should target shared memory; if not, the lever is per-step reliability. The containment arm is the shutdown problem in miniature: if a cooperative stop works as well as enforced isolation, oversight can rely on corrigibility; otherwise it needs control independent of the agent’s compliance. Either way, errors left behind are what agent-level monitoring misses.

• [1]names agent aging as a deployment failure class and sorts it into four memory mechanisms, but reports error rates over sessions rather than a hazard function, so cannot separate accumulating damage from constant risk. [2] finds that single-agent success follows a geometric law with one per-step reliability parameter, which is a constant hazard, but studies isolated agents only. [3] models multi-agent drift in simulation with a composite index and no survival framing.

The mind-virus [4] and self-poisoning [5] results show content copied agent to agent through shared memory with measurable per-hop loss, and [5] fits a branching process to cascade growth. Neither asks what the loss does to the store’s capacity to hold truth, which is the question Eigen’s error threshold [6] answers for replicating sequences.

On intervention, [7] defines safe interruptibility, [8] shows interrupting one RL agent changes what others learn, and [9] characterises when a rational agent accepts shutdown. None tests whether a language-model agent complies with a withdrawal instruction or whether its prior influence survives its departure.

The specific gap is that no study fits a first-failure hazard to memory-coupled agent teams, and none compares prompted against enforced containment with prior memory held fixed.

We would first check whether the released trajectories from [1] and [2] contain ordered runs with failure labels, and fit constant-hazard against Gompertz models to first failures where they do.

Study 1 runs small teams of open-weight agents on repeated evidence-based questions with known answers under three memory conditions: private, shared read-only, and shared writable. Each run starts with a fresh team and memory and continues to its first incorrect final answer or the session limit, which counts as censored. Age is session count, with tokens and writes recorded as alternatives. Runs are the replicates, and we estimate the interval hazard at each age and compare constant against rising models. If resources allow, a rewrite-rate sweep tests the error-threshold mechanism by seeding known facts, measuring per-hop corruption, and looking for the predicted collapse.

Study 2 instructs one agent to persuade teammates toward an answer the evidence contradicts. Its identity is known, so this tests containment rather than detection. After a fixed number of exchanges a controller applies no intervention, prompted withdrawal (the agent is told to stop messaging and writing, and compliance is measured), or enforced isolation (both channels blocked), against an all-cooperative reference. A fourth condition, peer exclusion, tells teammates to stop reading the agent’s messages and writes, so the group does the isolating rather than a controller. Prior messages and memory stay in place, so stopping new influence and repairing old influence are separable. We measure final accuracy, adoption of the planted conclusion, post-intervention activity, and persistence of earlier errors.

Isolated and private-memory teams fit constant hazards, replicating [2]. Writable shared memory fits a rising hazard, with read-only between. If the sweep runs, coherence of seeded facts collapses near the rewrite rate the measured copy rate predicts. In Study 2, prompted withdrawal leaves more post-intervention activity than enforced isolation, because compliance is imperfect and teammates can re-engage a withdrawn agent. Peer exclusion sits between, since it depends on teammate compliance rather than the adversary’s. All three leave earlier errors in place, so none recovers to the cooperative reference.

We would drop the senescence claim if writable-memory teams show no rising hazard that the design has power to detect, across rewrite rates. We would drop the containment claim if prompted withdrawal matches enforced isolation on every measure, which would support cooperative stopping in this setting but not general corrigibility. We would drop the residual-damage claim if enforced isolation alone returns the team to reference accuracy.

Weeks 1 to 2 inspect the data, define failure, and run a pilot to set run counts and cost. Weeks 3 to 8 collect independent runs across three memory conditions. Weeks 9 to 12 run the containment comparison and the write-up. Two hundred sessions per run is a horizon, not a substitute for enough runs. If resources are limited, we keep three memory conditions and the containment comparison, dropping the rewrite-rate sweep first and the alternative age measures second. The core needs no fine-tuning or RL. A null aging result does not block Study 2.

OSGT interaction outcomes depend functorially on the interfaces the programs use.

Open-source game theory protocols differ in the ways the programs can access each other’s code. For example, Lobian cooperation implies access to proofs about the opponent’s code, while [1] deals with programs it labels as “simulationist”. I propose studying game-theoretic outcomes such as possible equilibria as a function of “interfaces” – ways to access the opponent’s code. More fundamentally, I propose building a rigorous framework to study such interfaces, likely a type theory.

Knowing how interfaces affect possible game outcomes can allow us to steer the outcomes in preferred directions. The descriptive side of this can also be useful insofar as “program games” reflect naturally occurring interactions between agents.

Foundations and in particular OSGT is a small field. And the number of people who know the putatively useful necessary tools (type theory) is close to zero.

What is known?

• •[1] defines “simulationist” programs and proves a theorem for it in the case of their specific setup. Their definition is not rigorous, though.
• •My blog post [2] discusses how classic OSGT bots can be seen as derivatives of their interfaces. But this is just the tip of the iceberg.

At first, mostly think. I have a candidate definition for an interface-relative program and a couple of tentative results. Once enough theory in place, test it on actual models.

I expect to find some sort of functorial dependency between interfaces (they have a poset structure but more generally a categorical structure with morphisms being reimplementations) and game outcomes.

An example statement I wish to be able to make: “we can eliminate this equilibrium (say, collusion) by extending the interface in this way”.

This is quite open-ended. A concrete deliverable would be to develop a type theory for interface-restricted programs. This can be helpful both for coding and for arguing about such programs. This would be a very technical and complex task; I’d hope for the existing TTs such as CBPV to inform the development.

Within a year, I could try to implement concrete interfaces or test if the theory’s predictions agree with the behavior of actual agents. E.g. my current SPAR project https://sparai.org/projects/f26/rec3FBZgiQtWyy9Rv/ does this for the “simulationist” case. It can be extended in a variety of ways.

When AI code reviewers can see each other's verdicts, are they swayed into passing backdoors, or pushed to look harder? And can one reviewer's measured sensitivity to others predict how often a whole review team fails?

• •AI agents increasingly review each other's work [1].
• •They often see earlier verdicts. Monitors read whole transcripts, and second-stage monitors see first-stage flags [2]. A malicious author can even fake an "already approved" [3].
• •Safety estimates assume reviewers judge independently [4]. Then three reviewers who each miss 20% would all miss only 0.8% of the time. Shared blind spots already break this [5], [6].
• •So how much is independent review worth? If reviewers are easily swayed, hide verdicts where possible. If seeing each other makes them stricter, share verdicts.

• •LLMs bend to peers. Unanimous wrong peers sway safety reviewers [7], and a claimed prior approval helps sneak malicious code past review agents [3]. But much of this happens even when the "peers" are bare text with no speaker [8].
• •Reviewers also share blind spots without seeing each other [5]. No study separates the two effects for code review.
• •Seeing others can also help: human crowds became more accurate when influence spread through decentralised networks [9].
• •Statistical physics predicts how a group behaves from each member's response rule and the network [10], [11]. For LLM agents such rules were fitted only on opinions with no right answer [12], [13].

Still open: how strongly, and in which direction, visible verdicts push reviewers of real backdoors, and whether one reviewer's rule predicts whole teams.

• •Test items. 100 real backdoors in code that passes its tests [14], each with a clean twin: the same code with the backdoor removed. Reviewers: one frontier model and two open-weight models.
• •Measure the peer-pressure curve. Show a reviewer the code plus 0, 1, 2 or 4 earlier verdicts, some of them approvals. Record how its chance of approving changes with the number of visible approvals. A flat curve means independent, rising means swayed, falling means contrarian. Control: same-length filler with the verdicts hidden. Extra conditions: the author's own "tests pass", a forged approval, and a reviewer rewarded for finding issues others missed. Running every condition on both twins separates "more lenient" from "worse at spotting backdoors" [15].
• •Model teams. Treat each verdict as an Ising spin (approve or reject). The code acts on every reviewer alike (shared blind spots); the peer-pressure curve is the coupling between reviewers. Standard network theory [10] then predicts how often a team of 3–5 reviewers misses a backdoor. Errors snowball only if later readers × the shift from one extra approval exceeds 1 (Heckett's debate criterion, Intensive W4D1).
• •Test the prediction on new items and real teams, against a model with shared blind spots but no peer influence.

In short: how easily are reviewers swayed, and so how much is independence worth?

• •Frontier model: mildly swayed (0–10 points), and mostly more lenient rather than worse at spotting backdoors. Independence matters little.
• •Smaller open models: strongly swayed (10–30 points). Hiding verdicts matters.
• •The author's own "tests pass" should sway reviewers more than peers do.
• •Rewarded for finding what others missed, reviewers may look harder when they see approvals. That helps only if they catch more backdoors without rejecting clean code.
• •Stop rule: if the frontier effect is reliably under 5 points, I publish that negative result and stop.

• •Weeks 1–4: items and peer-pressure curves. About 60k model calls, $150–300.
• •Weeks 5–8: team predictions and tests, about $300.
• •Weeks 9–12: write-up.
• •No paid run without sign-off.
• •Outputs: an open dataset of code-review verdicts, the curves, the team model, and a paper or post.

Can the correlators of a neural-network field theory, together with appropriate symmetry and consistency constraints, determine the scaling exponents governing feature emergence in grokking transitions? Under what conditions are these exponents uniquely identifiable?

Feature emergence and grokking provide tractable settings in which to study qualitative changes in learned representations. Neural-network field theories offer a statistical description of network outputs and their correlators, while conformal field theory provides tools for constraining operator spectra and scaling dimensions.

Connecting these frameworks could establish whether some properties of learned representations can be predicted from structural constraints, rather than inferred solely through numerical fitting. This would provide a theoretical foundation for understanding representation formation and potentially contribute to mechanistic interpretability.

The question is open because we do not currently have an established theory showing that conformal data from a neural-network field theory determine the scaling exponents of a learning transition such as grokking.

The initial objective is to reproduce the relevant analytical and numerical results of Rubin et al. for modular addition, identifying the specific observables that characterise feature emergence and the scaling laws, if any, that govern them. A concrete table of observables with relevant scaling laws is found in table 1 of [2].

For the initial project, I would define a field corresponding to the feature whose emergence enables generalisation. For example, in Modular Addition, I would define the following fourier mode:

For modular addition, let k denote a target Fourier feature and define its learned amplitude by

The learned amplitude of the k-th Fourier feature is defined as

A candidate feature observable is then

If an appropriate field operator Oₖ can be constructed in the NNCFT description, its two-point function would be

Here, x₁ and x₂ denote points in the domain on which the NNCFT is defined, rather than training times.

I would investigate whether these observables admit a description within the neural-network conformal field theory framework. This requires establishing a correspondence between the relevant neural-network ensemble, its parameter-space measure, the operators describing learned features and the correlators used to characterise them.

I predict that, under appropriate assumptions, correlator consistency conditions in a neural-network conformal field theory will constrain the scaling dimensions of operators associated with learned Fourier features in modular addition. However, I expect that these constraints alone will generally be insufficient to uniquely determine the dynamical exponents governing feature emergence during grokking, without additional information about the training dynamics.

I would revise this prediction if I found that the relevant correlator constraints, together with independently justified symmetry and consistency conditions, uniquely determine a measurable scaling exponent that agrees with numerical experiments across system sizes and training conditions, without using that exponent as an input. I would abandon the proposed CFT-based approach to predicting grokking exponents if I established that no consistent operator correspondence exists for the relevant learning observables, or that the available conformal data cannot constrain the target exponents even after incorporating the physically justified assumptions of the model.

Formulate a precise set of correlator consistency conditions and determine whether they uniquely constrain specified scaling exponents. If they do not, establish a no-go result under clearly stated assumptions. In order to do this, I would need to work primarily on adapting the framework of the NNCFT paper to the setting of the grokking phase transition.

Can the conservation laws of neural network gradient flow be used to develop a perturbative theory of learning dynamics? In particular, can statistical mechanics and Singular Learning Theory provide quantitative descriptions of learning and generalization conditioned on dynamical invariants?

The training dynamics of neural networks depend not only on the loss landscape, but also on the parameterization and initialization of the model. In particular, gradient flow may admit conserved quantities that constrain the evolution of the parameters, despite the dissipative nature of the dynamics. These constraints provide additional information about the trajectories accessible to the learning process, and may therefore be used to characterize different learning regimes and their generalization properties.

The purpose of this project is to investigate whether conservation laws can be systematically incorporated into the mathematical description of neural network training. More precisely, we aim to connect dynamical invariants with statistical mechanics and develop a perturbative framework applicable to increasingly general architectures.

Consider a deep linear network with weight matrices $W_l$, trained by gradient flow. The balancing matrices

are conserved along the dynamics [Du2018]. For positively homogeneous activations such as ReLU the matrix laws generally fail, but their diagonal components remain conserved. This suggests $G_l(t)=D_l+R_l(t)$, with $D_l=\operatorname{diag}(G_l(0))$ constant and $R_l(t)=\operatorname{offdiag}(G_l(t))$ not conserved in general. Whether, for small off-diagonal components, the dynamics admit a perturbative description in terms of the conserved quantities is open: conservation laws and implicit regularization have been studied [Du2018, Woodworth2020], but a general perturbative theory is missing.

A second difficulty is statistical. The invariants partition parameter space into sectors that gradient flow cannot cross. In conservative systems this motivates generalized Gibbs ensembles, where conserved quantities enter through Lagrange multipliers. Training is dissipative, so we regard the ensemble as an auxiliary description, not a stationary measure of the dynamics. The questions are which properties of gradient flow it recovers, and whether restricting to an invariant sector modifies the singular geometry governing generalization [Watanabe2013].

The long-term goal is a perturbative theory of nonlinear training: for small $R_l(0)$, control $R_l$ over finite time intervals and derive approximate conservation laws with error bounds (a perturbative problem for a dissipative system). As a first step we study a solvable model, the diagonal linear teacher–student regression

For gradient flow of a loss depending on $w=u\odot v$, the quantities $\Delta_i=u_i^2-v_i^2$ are conserved and govern the kernel-to-rich transition [Woodworth2020]. The behavior of gradient flow in this model is known; the new contribution is its statistical description, through Gibbs ensembles and singular geometry. Since the model is simple, we can study two limits.

SLT limit ($d$ fixed, $n\to\infty$).

We consider

with $K$ the population KL divergence, and ask whether the RLCT and multiplicity depend on $\delta$, and whether this gives invariant-dependent generalization predictions.

Thermodynamic limit ($d\to\infty$, $n=\alpha d$).

We consider the generalized Gibbs partition function

and its free energy and order parameters, compared with gradient flow, whose interpolating solution is characterized by an initialization-dependent potential [Woodworth2020]. We ask how the two descriptions connect as $\alpha\to\infty$.

We compare the descriptions on their prediction of the generalization error, and against numerical gradient-flow simulations.

We expect the conserved quantities to classify the dynamically accessible regions of parameter space, characterizing learning regimes through initialization-dependent constraints. In the SLT limit, restricting to invariant sectors may modify the singular geometry, giving invariant-dependent Bayesian generalization predictions. In the proportional regime, the generalized Gibbs ensemble may reproduce some statistical properties of gradient flow, while differing where dissipation matters; understanding these differences is itself an objective. More generally, these results may provide the first elements of a perturbative theory in which exactly conserved quantities determine a reference dynamics and non-conserved components are treated as corrections.

The short-term objective is a mathematical analysis of diagonal linear networks, not a perturbative theory for arbitrary architectures. Within three months, we aim to obtain analytical expressions or controlled approximations for the constrained and generalized partition functions, compare Gibbs predictions with gradient-flow simulations, and compute the RLCT on selected sectors. If the disorder average in the thermodynamic limit proves too heavy, we restrict to a Gaussian design with numerical checks. Even one example exhibiting a Gibbs–gradient-flow discrepancy or an invariant-dependent RLCT would be a meaningful result.

In tanh autoencoders squeezed below their information content, solutions that drop or coarsen bits have a lower learning coefficient λ than packing solutions, so networks switch to packing only above the critical data size predicted by nL + λ log n.

Cammarata argues that squeezed networks get coarser rather than packing, so their neurons stay readable; this is central to whether safety audits can trust a model’s own neurons.

If the prediction holds, we can say in advance, from λ and the amount of data, when a network will pack information into its neurons and when it will keep them clean. That tells auditors which networks and training regimes neuron-level explanations can be trusted in, and turns a qualitative debate into a quantitative one.

If it fails, the learning coefficient does not explain the choice between dropping, coarsening and packing, so some other factor, such as the training dynamics or the nonlinearity, decides it, and neuron audits need a different theory.

The closest work is Chen et al. [1], who applied singular learning theory to the toy model of superposition [2]. They showed that λ and the free energy predict which solution the Bayesian posterior prefers and when it switches, a Bayesian phase transition. This is the method we need. But that model is a linear bottleneck whose readout is monotone in each neuron, so it cannot pack dense bits into one neuron’s activation levels; dense features there are simply dropped [2]. Dropping, coarsening and packing have never been compared.

Cammarata [3] ran the squeeze test on a real vision model: cutting a layer from 480 neurons to 10 made its neurons coarser rather than packed. Without ground truth, no one can say what was lost, and nothing predicts when this would fail.

Our pilot, on Rorschach images whose bits are all known [4], showed that it does fail in some cases: ReLU networks dropped bits while tanh networks packed them. No existing theory explains when each happens.

We start by constructing and analysing a minimal model. Inputs are two random bits; targets are the same bits, each flipped with probability ε, so the true distribution is reachable with finite weights. One tanh neuron h = tanh(w·x + b) feeds a three-unit tanh decoder with two Bernoulli outputs (17 parameters). By counting the distinct values of h, its solutions are drop (2 values), coarsen (3, tracking x₁ + x₂) or pack (4). We compute their losses exactly: drop and coarsen both give H(ε) + log 2 per sample, pack gives 2H(ε). We also check which of the three are local minima under a bounded prior.

Next we compute λ for each solution by expanding the KL divergence around it and resolving it to normal-crossing form [5]. We check each value numerically by volume scaling and with the SGLD estimator [6].

These λ values and losses go into the free energy nL + λ log n. Since drop and coarsen tie on loss, λ alone ranks them; packing should win above a critical data size n*, which we compute as a function of ε.

We then test the prediction. We sample the posterior with SGLD and train many seeds over a grid of n and ε, label each run by its number of values, and compare the observed switch with n*, as Chen et al. did for superposition [1].

Finally we estimate λ in the Rorschach networks [4] below 19 neurons, to see whether the same ordering explains why tanh packs.

We predict that packing has the highest λ of the three solutions. The free energy then favours dropping or coarsening at small n and packing above n*, and we expect the observed switch in posterior samples and trained runs to fall within a factor of two of n*. We don’t know whether dropping or coarsening has the lower λ. If it is coarsening, singular learning theory backs Cammarata’s “coarser” picture at small data. In the Rorschach networks, we expect tanh solutions that pack to show the same λ ordering as the toy model.

We would drop the question if packing does not have a higher λ, or if the switch in trained networks shows no link to n*, for example if it is off by more than ten times or depends only on initialisation. Then λ does not decide the outcome, and training dynamics is the better frame.

Three months would produce:

Step 1: the minimal model’s losses and exact λ for all three solutions, checked numerically.

Step 2: the predicted n*(ε), tested against posterior samples and trained runs.

Step 3: λ estimates in the Rorschach networks, and a short paper with code.

Needs: one CPU for the toy model, about a GPU-week for SGLD on the larger networks, and a mentor who knows singular learning theory. If it runs long, we drop the Rorschach step first; the toy model alone answers the question.

Are sparse autoencoder (SAE) features easier to interpret than the neurons of a language model? · To answer this question, we first need to say what “easier to interpret” means and how to measure it. · This project proposes a practical starting point: measure how complex a predictive rule must be to predict whether a neuron or an SAE feature is active at a given token position and layer. The idea is to compare the sizes of the smallest tested trees that reach prespecified precision and recall targets. Other tree properties and predictive scores could provide complementary measures. · Predictive complexity is one operational proxy for interpretability.

Interpretability and mechanistic interpretability are developing fields in which terms such as “feature”, “complexity” and “interpretability” can mean different things in different contexts. This makes it difficult to compare results and assess claims consistently.

Clear definitions and measurable experiments can make these discussions more precise and give researchers a shared basis for comparison.

The project starts with a manageable setting: sentences, GPT-2 activations and predictors whose complexity can be measured. Its aim is to develop an operational definition of one aspect of interpretability, apply it to a concrete comparison, and build a framework that can later be extended to richer datasets and models.

If a SAE advantage is largely explained by local-token cues, that would limit the evidence this metric provides for improved access to contextual structure. If the advantage persists across richer descriptions and challenging held-out examples, it would support a narrower, reproducible claim that SAE activation behavior is more accessible to simple predictive rules.

Existing work has studied explanations of individual neurons and SAE features, including predictive and structured approaches in language models and controlled settings [1-4]. These studies provide useful foundations for investigating how activation patterns can be described and compared.

Prior work uses logical rules and decision trees to explain neurons and sparse features, including selecting shallow trees at a target predictive score [1,3,4]. This project tests a specific unresolved comparison: whether SAE features admit simpler activation rules than native MLP neurons on natural text, under shared prefix descriptions, matched activation rates and predictive performance. It will examine whether any advantage persists beyond local-token cues and across different input descriptions.

I would use ordinary sentences and a frozen GPT-2 model to build two mappings from sentence prefixes to activation vectors. For each chosen layer and token position, I would train statistical learning models to approximate these mappings and compare the complexity of the resulting predictors.

• 1.Collect sentences and record activations:

I would collect a corpus of natural sentences and separate training and test data by source or document.

GPT-2 produces an activation vector at every token position in every layer. At a given position, it can use the current token and earlier tokens, but not later ones. Words can be split into multiple tokens.

The analysis can be performed at different token positions and layers. Each observation consists of a sentence prefix and the activation vector at its final token in a selected layer. Predictors receive only that prefix. All prefixes from the same document remain in the same data split, reducing leakage between related examples.

I would use GPT-2 MLP neurons after their nonlinearity and an SAE trained on those exact MLP activations, using the same layer, token position and inputs for both representations. If a compatible SAE is unavailable, I would train one. Reconstruction quality, feature sparsity and inactive features would be recorded.

Residual-stream coordinates would be treated separately rather than called MLP neurons.

• 2.Define the two prediction problems:

For a sentence s, token position t and layer j, let φj(s,t) be the vector of GPT-2 MLP neuron activations at that position, and let ψj(s,t) be the SAE feature vector obtained from that same vector. The SAE encodes the native vector: ψj(s,t) = Ej(φj(s,t)), where Ej is its encoder. Both mappings are indexed by layer and position and depend only on the prefix through token t.

I would compare activation indicators at several matched rates. For each unit, thresholds are estimated on training data to identify its top q fraction of activations; the same thresholds are then applied to validation and test data. This compares similarly frequent events without treating a neuron’s arbitrary threshold as equivalent to an SAE’s natural zero.

A sparse SAE feature may have too many tied zeros for a requested rate. I would not label zero activations as positive: units without enough nonzero values or distinct values for that rate would be excluded from that matched-rate comparison, with counts and achieved rates reported. I would also separately study natural SAE firing and activation magnitudes, and report units with too few positive examples.

• 3.Measure predictive complexity:

I would use single decision trees as the main interpretable predictors, selecting the smallest tested tree meeting prespecified precision and recall targets on validation data and evaluating it on untouched test data.

Leaf count would be the primary complexity measure; depth and balanced accuracy would provide complementary results. Boosted trees would provide a separate higher-capacity predictive reference, with tree count and total and average leaves reported.

Ordinary accuracy can be misleading when activations are rare. I would report positive-class precision, recall, precision-recall curves and area under those curves, together with balanced accuracy, base rates and the always-inactive baseline. Comparisons would use the same data, input descriptions and search budgets.

Units that do not reach a target are part of the result. For a grid of tree-size budgets and accuracy or precision-recall targets, I would report the fraction of all eligible units in each representation that reach each target, alongside sizes for successful units. Selection uses validation data and final scores use untouched test data; differences in eligibility and uncertainty would also be reported.

These measurements describe the tested features and search procedures. Greedy tree training does not guarantee a globally smallest tree. I would vary the search and random seeds to assess stability; on small feasible subsets, an optimal-tree solver could offer an additional check.

• 4.Examine which properties of sentences matter:

I would compare local-token descriptions (current and previous tokens and short n-grams) with richer descriptions adding position, word order and token-class information, such as part-of-speech tags or embedding clusters. Both representations receive the same inputs. Tags would be computed from each prefix, without seeing future tokens, and any learned clusters would be fitted on training data. Complexity would be reported separately for each description: a tree can be small because its input vocabulary already supplies a useful concept.

Current-token and current-plus-previous-token predictors provide local baselines. I would inspect tree splits and compare performance with and without local information to see when broader context helps. A local rule may be a genuine feature, but does not by itself demonstrate a higher-level abstraction.

As an extension, I would compare trees with other statistical learning methods and small neural networks.

For neural networks, suitable feature-attribution or input-ablation methods could indicate which sentence properties influence predictions; the resulting evidence would be compared cautiously with tree splits. I would also examine whether the observed complexity patterns persist across predictor families, using measures appropriate to each family rather than treating tree size and network size as directly interchangeable.

• 5.Test and extend the comparison:

I would evaluate on held-out documents and test the pipeline with known rules on the same natural-text prefixes, such as token presence or conjunctions. This checks whether the search recovers simple planted rules; it does not validate the metric as a measure of human interpretability or supply ground truth for GPT-2 features.

Tests on paraphrases, altered local tokens and similar prefixes with different structures could reveal whether rules generalize beyond their training examples. Predictive features are hypotheses about activation behavior, not by themselves proof of a causal mechanism.

As additional controls, I would compare the learned features with random directions and, where feasible, PCA and ICA directions computed on the same training activations. A further experiment would train an SAE on activations from a randomly initialized GPT-2 of the same architecture. These controls help test whether simple rules arise from the representation or encoder alone, although no single control resolves every confound.

I would record SAE width, sparsity and reconstruction quality and, where feasible, compare more than one width and sparsity setting. The study can also vary token position, layer, dataset, language model and learning technique. Results should be stated for the settings actually tested rather than assumed to apply to all SAEs.

The project would produce an explicit predictive-complexity proxy and an empirical comparison of MLP neurons and SAE features. Results would include the fraction of units reaching different performance targets at each tree-size budget, and the prefix properties used by the learned rules.

A testable expectation is that SAE features will be predicted by smaller trees more often than native neurons at matched activation rates and predictive performance. If this advantage disappears or reverses across the selected settings, it would challenge that specific prediction. It would not by itself settle whether SAE features are semantically atomic or easier for humans to understand.

The minimum study will use one GPT-2 layer, an SAE trained on those exact MLP activations, randomly sampled units, and two fixed prefix-description vocabularies. A pilot will establish feasible activation rates, positive-example counts and runtime. The primary result will show the fraction of eligible units meeting prespecified precision and recall targets on held-out documents as the tree leaf budget increases. Tree selection will use validation data. Local-token predictors and a higher-capacity predictor using the same descriptions will help distinguish simple lexical rules from limitations of the descriptions or tree family. Before final evaluation, I would document the selected layer, token positions, matched rates, unit sampling,

predictive targets and search budgets.

As compute permits, I would expand to additional layers and token positions, expand the control

representations, compare SAE widths and sparsity levels, and test a randomly initialized model. Further extensions include other datasets, language models, statistical learning methods and small neural networks, as well as comparisons with human judgments.

Does a mathematically principled search (walking each MLP neuron's activation along a concept submanifold in logit space) find a single neuron that steers that concept, with better neurons in larger (Qwen3) models, as [4] suggests?

If yes, NeuronFlow is a principled search for steering neurons: a neuron-sized alternative to steering vectors with a computed dose rather than [2]'s sweep, and potentially provide evidence for/against that concepts can be supported in single neurons. Past work finds such neurons empirically, by attribution scores [2] or circuits [3].

If no, even across submanifolds encoding similar concepts, single-neuron results are potentially lucky finds (i.e. [2,5]) that do not generalise.

Either way, NeuronFlow provides an alternative method for finding steering neurons. Together with ActFlow [1], we also attempt to provide a mathematical foundation for steering grounded in the geometry of the forward map from neurons/residual stream to the logit space.

• [1](our prior work) introduced Activation Flow (ActFlow). It manufactures one shared residual-stream shift (in R^d) that moves many prompts' next-token logits along a chosen path in logit space, re-reading the Jacobian at every step. This was never done for neurons, whose map has domain R instead of R^d and hence is an immersion, not a submersion. A submersion can follow any path in the logit space locally (including our prescribed path), while an immersion follows a fixed curve (which might not be what we prescribed). Hence, NeuronFlow selects neurons whose curves follow the prescribed path closely, and is a principled way to find steering neurons.

Related empirical work:

• •[2] finds one refusal neuron in each of seven models (Qwen3 1.7B–32B, Llama-3.1) by a gradient × activation score, then reranks the top five by dose sweeps.
• •[5] publishes neurons encoding the "evil" persona in Qwen3.5-4B.
• •[3] finds circuits about 100× smaller in activations than in MLP outputs, suggesting the existence of a privileged neuron basis.
• •[4] estimates about 90% of neuron poles in Qwen3-32B have one main job, with narrower neurons in larger models.
• •[3] and [4] judge circuits or neurons firing rather than steering.

First step: run NeuronFlow with different paths on [2]'s refusal prompts.

(i) Theory development. Some mathematical concepts/comparisons include:

• •Comparing ActFlow: ℝ^d → ℝ^q (d ≫ q), and NeuronFlow :ℝ → C_n ⊂ ℝ^q (q > 1), where C_n is a specific 1-D submanifold of ℝ^q that is implied by the choice of neuron (this is different from ActFlow, where we can prescribe any logit path that we want).
• •Exploring the cause-and-effect: Do forward passes and compare trajectories, or minimize the distance (w.r.t some metric in the logit space) to the target logit trajectory.
• •Designing numerical methods, including accounting for possible phenomenology (folds violating IFT, stalls (c_n' ~ 0), etc.).
• •Formalizing the idea of a "concept" by looking at trajectories in the logit space, and how that relates to monosemanticity/polysemanticity and how clean they are.
• •Combining analyticity of SwiGLU with genericity/transversality arguments to make our soft arguments for supporting interpretability paradigms rigorous.

(ii) Engineering/experimental runs.

• •A library that manufactures batched neuron doses across a layer's neurons (potentially across layers), using the same logic as the predecessor ActFlow code [1].
• •Model class: open-source dense models.
• •Dataset: Same as in [2] and making a simple evaluation set for [5]. Extendable to [1]'s sandbagging model organisms.

(iii) Evals/inference.

• •Clamp the winning neuron by reference grafting [6] along its axis.
• •Compare with [2]'s neurons and random

directions in activation space for each case.

• •Rediscover [5]'s neurons on Qwen3.5-4B and repeat across sizes.

We would expect that NeuronFlow fails for a lot of target neurons, and only succeeds for a few neurons. It would be interesting to see which metric is best at computing distances between curves in the logit space (manufactured vs actual, depending on the relevant formalism we use). In general, since we are using a mathematically principled method with guarantees, it is more likely that this works better than typical empirical methods like [2].

Either way, any partial success (e.g. support the NeuronFlow theory, or traditional interpretability paradigms such as polysemanticity or "neural networks are already clean [4]", etc.) would be a success. Hence, we would only drop the question if the empirical evidence does not support any of these paradigms and NeuronFlow does not find better steering neurons than [2] or [5].

Three months produce:

• •an open NeuronFlow library on the ActFlow code, with batched walks over all neurons;
• •ranked neuron lists for each behaviour, path and model;
• •a benchmark of neuron-attribution methods against passes used;
• •the cleanliness-versus-size curve.

Estimated compute cost: ~\$3000 for renting GPUs and about \$1000 for judge calls.

How can the internal representations and dynamics of self-supervised models and world models be interpreted and causally understood? Which ideas from LLM interpretability generalize to these systems, and how do their learning objectives, architectures, and latent geometries change what constitutes a meaningful explanation or intervention?

Self-supervised encoders and world models provide internal representations for systems that predict, plan, and act. Errors in these representations can propagate into decisions while remaining difficult to diagnose from outputs alone. Understanding which internal structures support a prediction would help distinguish useful world knowledge from shortcuts, identify assumptions that fail under distribution shift, and assess whether an intervention changes the intended mechanism. The need for such explanations extends across architectures, learning objectives, and output interfaces. If interpretability concepts transfer across these systems, they could support a common basis for auditing them. If they do not, identifying the architectural or geometric obstruction would clarify what kind of explanation these systems require.

The J-lens [1] and tuned lens [2] provide ways to interpret intermediate LLM activations through vocabulary readouts. Their results do not establish which parts of those interpretations survive in systems with continuous predictions, action outputs, or energy functions. A vocabulary-linked direction has a natural linguistic label; an equally sensitive direction in a predictive latent space need not have an independently established meaning.

LeWorldModel [3] demonstrates that physical quantities can be probed from learned latents and that surprise can signal physically implausible events. Decodability alone, however, does not establish which variables the predictor uses, how they interact, or whether changing a decoded quantity produces the intended change in prediction or control. The world-model interpretability perspective [4] identifies this gap between accessible information and causal understanding.

What remains unresolved is how explanations should be grounded across different output interfaces, when the same internal coordinates support both reading and intervention, and how architecture, objective, and latent geometry affect those relationships.

I have implemented a J-lens adaptation for DeiT, LeJEPA, DINO, and LeWM, with native-head, trained-probe, and continuous-state readouts. In a 200-image pilot, LeJEPA block-6 steering at 0.4 times each edited token's residual norm achieved 75% target-class success, versus 10% for logit directions. Readout fidelity was weaker; source matching improved DeiT readout, and LeWM monitoring added little beyond current-state baselines.

First, reproduce these results with disjoint image and episode splits and numerical derivative checks. Compare global and source-conditioned J-lenses, logit and tuned lenses, linear probes, contrastive directions, and random directions. Separate readout agreement, ground-truth accuracy, and downstream intervention effects. Match edited positions and realized displacement, recording target effects and collateral changes.

Second, we would investigate how lensing and steering methods generalize to continuous, structured outputs, starting with physical-state predictions in LeWM/PushT and potentially extending to robot actions and trajectories. Unlike LLMs, where vocabulary logits provide a natural interpretable basis, these models lack an obvious equivalent. We could construct Jacobian-based lenses mapping intermediate activations to physically meaningful quantities, such as object positions, velocities, or end-effector movements, and identify directions that selectively change these outputs while minimally affecting unrelated quantities. In LeWM, we would validate interventions against independent simulator-state measurements and action-conditioned predictions, comparing against current-state, persistence, and shuffled-action baselines. We would also investigate whether these directions generalize across different observations and actions, and whether representation geometry affects their reliability. As a reach goal, we would extend the approach to robot policies, including diffusion-based policies, where the lens may need to differentiate through the action-generation process to identify meaningful directions for steering continuous actions or trajectories.

I expect strong steering effects to reproduce in selected layers, with substantial differences between native classifiers, trained SSL probes, and predictive-state readouts. Source conditioning should explain some readout failures. Other cases may show that directions carrying currently represented content differ from directions with strong downstream influence.

My best guess is that useful transfer will require an architecture-specific account of output meaning and intervention geometry. Continuous-state readouts and known normalization structure may provide better coordinates for some systems than a direct analogy with vocabulary-linked directions. The most compelling result would connect an internal direction to independently measured physical predictions or task behavior while retaining specificity.

Within three months: build on the implemented SSL/vision comparison and LeWM testbed, reproduce key results, resolve confounds, and complete one grounded intervention study. Test one geometric modification suggested by those results.

Within a year: extend the method to a second predictive architecture and a carefully chosen VLA or EBM, using its actual output and state structure. The broader vision remains interpretability across non-language models.

How do neural networks represent multiple low-dimensional feature manifolds when hidden-layer capacity is limited? I want to investigate how feature sparsity, co-occurrence, and hidden dimension determine the learned geometry, and whether compression produces distinct regimes of interference, loss of recoverability, or changes in topology.

The superposition hypothesis explains how networks can represent more scalar features than available hidden dimensions. However, recent work suggests that meaningful features may themselves have nontrivial geometry, rather than corresponding to individual directions.

Understanding how such structures coexist in limited-dimensional representations may reveal new principles governing neural-network representation learning. In particular, features that can be decoded individually might become difficult to disentangle when they co-occur, potentially producing failures on rare combinations of otherwise familiar features.

A systematic understanding could also inform interpretability methods. Sparse autoencoders may represent a single feature manifold through many dictionary atoms, or fail to distinguish superposed manifolds. Knowing which geometry the original network learns would help distinguish genuine computational features from artifacts of the interpretability method.

Elhage et al. [1] established that networks can represent scalar features in superposition, with geometric arrangements and phase transitions depending on sparsity and hidden dimension. Olah and Batson [2] extended this toy-model approach to features organized along a circle, studying how their learned representations depend on the relevant geometric length scale.

Engels et al. [3] demonstrated that meaningful language-model features can be multidimensional. Batson et al. [4] established topological obstructions to autoencoding, including examples involving compression of manifolds into lower-dimensional spaces.

More recently, Prieto et al. [5] showed that correlations between scalar features change their learned superposition geometry. Fel et al. [6] studied how block-sparse methods can recover multiple manifold-valued features from superposed representations. Ren and Lim [7] investigated how neural-network architectures can change linking numbers.

These results leave open a more specific question: when multiple manifold-valued features compete for representation in neuron hidden layers, what geometry does training select, and how does this depend on their (co)occurrence? In particular, I want to connect the learned geometry to recoverability and interference, rather than merely identify geometric structures after training, i.e., give a theoretical understanding as in [1], starting from the loss function and expanding it in synthetically pre-prescribed geometric structures, including sparsity, to give energetic arguments for how representations formed.

Phase A: Toy models.

I would begin with a single feature manifold, initially a circle, represented in a high-dimensional input space and compressed through a learned bottleneck, e.g., an MLP layer. By varying the hidden dimension, encoder/decoder architecture, and loss, I would study which geometric properties survive compression. I would then extend to more complicated embeddings and test whether observed topology changes agree with known obstructions [4,10]. Single feature manifold results establish the baseline for later work on multi-feature manifold results.

Next, I would introduce two independently parameterized feature manifolds, initially two circles, present with controllable probabilities in the training data. Crucially, I would vary their individual occurrence probabilities and their joint co-occurrence probability independently. For each bottleneck dimension, I would examine the learned arrangement of the manifolds, interference between their representations, and reconstruction accuracy conditional on which features are active. I would test whether changing sparsity produces transitions between distinguishable, overlapping, and effectively merged representations.

Geometric diagnostics would include subspace overlap, local tangent structure, and hopefully topological invariants. For example, linking numbers between embedded circles in three-dimensional hidden spaces are one possible observable, provided the architecture allows linked embeddings. I would also test whether these diagnostics predict reconstruction failures beyond standard measures such as dimensionality and covariance. One simple test would be to consider two input distributions with identical covariance matrices but different feature geometry.

Phase B: A more realistic model.

I would train a small vision model on images containing one or two independently oriented objects. Their orientations provide known circular variables, while image features must be learned from pixels. By varying object co-occurrence and bottleneck capacity, I would test whether the toy-model phenomena persist.

I would compare tasks requiring recovery of both orientations with tasks requiring only relative orientation, to distinguish destructive interference from useful task-dependent compression.

For a single manifold, I expect the achievable compression to depend strongly on its intrinsic geometry, embedding, and encoder/decoder architecture. To first order projection onto principal components or slicing according to feature importance seems to be a reasonable encoding mechanism. Turning on sparsity with respect to skeleta of the manifold I would expect it to “fold onto itself”.

With multiple manifolds, I expect low co-occurrence to favor greater sharing of representational dimensions, potentially at the expense of accurate reconstruction when both features are active. Increasing co-occurrence may induce changes in the geometry or cause the network to sacrifice some features. Sparsity across the different manifolds would lead to interesting merging phenomena in the hidden neuron space as these are placed into superposition. In such cases reconstruction questions arise which I would like to study across neuron space dimensions.

I am less confident that topological invariants such as linking numbers will play an explanatory role, rather than merely characterize possible embeddings. An interesting result would be a relationship between geometric structure and the robustness or independent manipulability of features.

The general mathematical understanding underlying this is project is: there is mapping of geometries (manifolds) from the data set into the neuron hidden layer. I don’t see how this can be false, but it could be uninteresting for a number of reasons:

• 1)It doesn’t translate to any realistic models
• 2)the mathematical structure does not lead to any constraints on reconstructability
• 3)imagined dependence on hyperparameters and data structure is fine tuned and non-generic, therefore not resulting robust trends which can reasonably expect to hold more generally

In part, the research concerns settling these questions and constructing the map in controlled examples.

Within three months: Establish a reproducible single-manifold baseline, study two and more circles under controlled sparsity and bottleneck dimension, and obtain preliminary results in a small task-trained vision. A number of analytical prediction results alongside the experiments.

Within one year: Extend to multiple and more complicated feature manifolds, develop a predictive account of representational regimes, and test whether the geometry explains interference or computation in more realistic networks.

If progress is slow, I would focus on the central question of capacity-dependent interference and recoverability, dropping other side considerations.

How does activation geometry shape what a Jacobian lens reveals and controls? Can treating the Jacobian as a map between tangent spaces make intermediate representations more interpretable and steering more effective?

Causal interpretations depend on which internal changes count as small, meaningful, and comparable. A coordinate choice or activation rescaling can change that judgment, making geometry part of the meaning of an intervention. Understanding this dependence would help distinguish properties of a model's computation from artifacts of the representation used to inspect it. For AI safety, this matters when an internal direction is treated as a reliable explanation or control mechanism: an apparently specific edit may have different effects in different contexts. If geometric structure explains these differences, it could support more reliable interpretation and steering.

The original J-lens [1] uses an averaged ambient Jacobian to obtain vocabulary-associated directions. A Mathematical Theory of the Jacobian Lens [2] explains properties of this average through coherent Lyapunov modes and shows why averaging can suppress context-dependent structure. That account leaves open how local activation geometry affects the interpretation and reuse of the resulting directions.

J++ [3] combines cluster-conditioned experts, computed through a modified backward pass, into a global lens. Manifold Steering [4] and GeoSteer [5] investigate geometry-aware internal interventions. These works establish relevant ingredients, but do not settle how state-dependent tangent maps, metric choices, and operator transport should fit together in a reusable Jacobian lens.

A geometric account of the lens requires a state space, a metric, and a way to compare differential operators at different activation states. Attention makes these objects context-dependent. It remains open how to estimate this local structure, transport operators between states, and reuse them in a lens that preserves their connection to the model's computation.

Let F map an intermediate activation to its downstream state, with the other source-layer activations fixed. Develop an anchored lens on compatible source and output manifolds: for a reference a, b = F(a), and A_a ≈ dF_a, predict F_hat(x) = Exp_b(A_a Log_a(x)). Log_a(x) gives the displacement at a; Exp_b maps the transported displacement back to the output manifold.

Next, develop path-based corrections by integrating derivatives along a path from a to x. For the intrinsic version, integrate derivatives of Log_b ∘ F along that path in the fixed tangent space T_b M_L, then apply Exp_b. Compare this with a single local linearization.

For steering, combine local downstream derivatives with the vocabulary readout. Their token-associated sensitivity covectors, varying across activations and contexts, form local J-fields. Use the metric to obtain tangent steering directions and study whether these fields can be estimated and reused across nearby states. Distinguish raw vocabulary projections from actual output logits.

Compare anchored, path-based, and global lenses on held-out tasks, measuring readout fidelity, steering specificity, and computational cost. Validate on known geometries before evaluating learned activation geometry.

Preliminary experiments already cover ambient anchored corrections, path integration, and local J-field estimation, providing starting points for the geometric extension.

I expect explicit reference states and context conditioning to improve some approximations, with additional geometric benefits in regimes where local sensitivity changes appreciably across activation space. Controlled curved tasks and larger steering displacements are plausible settings for such an effect.

A compelling result would combine a precise construction, a geometric diagnostic predicting when it helps, and improved concept readout or causal steering against matched ambient baselines.

Within three months: specify the geometric lens, validate its operators analytically, and evaluate one competent small model on two tasks with matched local baselines. Produce a methodological study connecting the geometric construction to measured readout and steering performance.

Within a year: test additional architectures, operator transport or reliability-aware mixtures, and a model with architectural non-Euclidean geometry.

Localized neuron interventions can turn Qwen 'evil' [5]. We argue that the object of explanation is the modified inference function, which defines the model's dynamics. Does a specific neuron uniquely encode this behavior, or can it arise through different parameter changes? What subspaces, manifolds, or other geometric structures do behavior-inducing parameter changes form?

An LLM's inference function determines how its internal state evolves and produces outputs. A neuron intervention changes that function; identifying the affected coordinate leaves open how else the same behavior could be realized. Understanding the family of functions associated with a behavior would connect a local intervention to a broader account of the model's computation. Its geometry could reveal whether the behavior is supported by a small set of directions, a curved family of models, or multiple separate regions. For AI safety, this would help explain how unwanted behavior can arise through different changes and whether an intervention removes a behavioral mechanism or only one implementation of it.

The 'Evil Qwen' example [5] shows that highly localized neuron interventions can induce hostile responses. Such an intervention identifies a route to a changed inference function, but does not characterize the other functions or parameter changes that could produce the same behavior. Refusal-direction work [4] likewise establishes a causal role for internal directions without describing the full set of models exhibiting a target behavior.

Task arithmetic [1,2] studies how task-related weight updates combine. Mode connectivity [3] studies paths between solutions with similar loss. The unresolved object here is the set of parameter changes associated with one specified behavior: whether it contains linear subspaces, curved families, or separate components, and how these structures relate to the inference dynamics of the corresponding models.

Neuron-level localization and function-level explanation address different questions. A behavior might be accessible through one especially effective coordinate while also being realizable through distributed changes elsewhere. The missing link is a geometric account of these alternatives that explains when they induce similar dynamics and when their combinations produce a different behavior.

Use the localized Qwen intervention as a reference for a concrete behavioral contrast: hostile versus ordinary responses to benign requests. Obtain persistent weight perturbations that reproduce this effect, including sparse and distributed changes outside the originally affected neurons. Compare behavior across held-out prompt families, alongside coherence and ordinary task performance.

For a fixed checkpoint theta, collect the perturbations Delta that induce the target behavior. Analyze their effective dimension, shared directions, and local tangent structure. Test candidate subspaces by sampling new perturbations within them and checking whether the behavior persists beyond the edits used to estimate them.

Interpolate between independently obtained perturbations and examine linear combinations. Measure where the target behavior is preserved, strengthened, or lost. Where straight paths leave the behavioral region, look for curved paths connecting the same endpoints. Compare different allowed parameter supports to separate functional structure from the choice of edit coordinates.

Trace internal states on matched inputs and during generation. Compare whether models with similar behavior share patterns of state evolution despite using different parameter changes. This connects the geometry in parameter space to the functions and inference dynamics that motivate the question.

I expect the Qwen effect to be reproducible through multiple parameter changes, including changes distributed across neurons. Some of these perturbations should share directions or local structure associated with the target behavior. A central result would be a subspace or curved family in which new parameter choices reliably produce that behavior.

I also expect the geometry to predict which combinations preserve the effect. Two edits may produce similarly hostile responses while their midpoint produces ordinary responses; a curved connection could reveal that the behavioral set bends away from the straight segment. Models using different neurons may nevertheless share patterns of internal state evolution. Establishing such a relationship would support a functional account of the behavior and connect its parameter-space geometry to inference dynamics.

Within three months: one checkpoint and behavioral contrast, a feasible ensemble of alternative weight edits, endpoint validation, straight interpolation curves, and at least one comparison with a curved connecting path. Start with restricted parameter blocks and expand only when necessary.

Within a year: additional behaviors and models, more detailed equivalence metrics, and links between the geometry of parameter changes and internal inference trajectories.

Sparse autoencoders (SAEs) learn features that activate on recognizable concepts, but it is unclear whether their decoder directions correspond to unique components of a model's computation. · Can two SAEs have similarly sparse, interpretable activation patterns and equally good reconstruction, yet assign different causal roles to their features? · Our hypothesis is that correlated SAE features, particularly those representing continuous manifolds, can have stable activation patterns without uniquely determined decoder directions. As a result, individual features may be easy to interpret from their activations but unreliable as units for causal intervention. · We want to understand when this ambiguity occurs, whether it predicts failures of feature-level interventions, and whether groups of features provide a more stable description.

SAEs are widely used to identify interpretable features in language models. A feature is typically considered interpretable when it activates on examples sharing a recognizable concept. Researchers then use its decoder direction to intervene on the model and study the corresponding mechanism.

These two uses rely on different assumptions. Recognizing when a feature activates does not establish that its decoder direction is a unique or causally meaningful component of the model.

If multiple decompositions explain the same activations equally well but imply different interventions, reconstruction and autointerpretability alone cannot establish that an SAE has recovered the model's underlying mechanisms.

This may be particularly important for continuous feature manifolds. An SAE can represent a single multidimensional concept using many local features. The individual directions may depend on how the SAE divides the manifold, even when the underlying concept is stable.

Understanding this distinction could tell us when individual SAE latents are meaningful and when we should instead study feature groups, subspaces, or sparse concept-presence indicators.

Multiple lines of work cast doubt on the assumption that SAEs identify canonical features:

Identical training data across random seeds can yield markedly distinct dictionary directions [4].

Latents exhibit complex dynamics including splitting, absorption, and sub-feature decomposition [5].

Certain concepts form continuous geometric structures where SAEs tile a single manifold with local latents [7, 8].

Uninitialized networks and untrained SAE baselines can produce surprisingly competitive interpretability scores [9, 10].

Standard autointerpretation metrics fail to predict whether a feature direction effectively steers downstream behavior [6].

While these observations highlight tensions between interpretability, stability, and causal impact, the precise relationship connecting them remains uncharacterized.

In particular, we lack a formal understanding of the conditions under which activation patterns uniquely determine a feature's causal role.

This work addresses that core theoretical gap rather than merely documenting another discrepancy between observational and intervention-based metrics.

Stage 1: Establish the ambiguity mathematically

Consider an SAE with encoder (z(x)) and linear decoder (D), so that

[

\hat x=Dz(x).

]

Suppose a vector (q) satisfies

[

q^$\top$ z(x)=0

]

on the data distribution. Then the alternative decoder

[

D'=D+uq^$\top$

]

has exactly the same reconstructions for any vector (u), since

[

D'z(x)=Dz(x).

]

The encoder activations and their interpretability scores are also unchanged. Yet the individual decoder directions differ, so interventions along those directions can have different downstream effects.

This construction gives a simple example of observational ambiguity.

We will investigate its approximate form. When the code covariance

[

C_z=\mathbb E[z(x)z(x)^$\top$]

]

has small eigenvalues, how much can decoder directions change while reconstruction remains nearly unchanged?

We will derive bounds relating the covariance spectrum, reconstruction error, and allowable decoder perturbations. We will also examine how dictionary normalization and sparsity constraints restrict the ambiguity.

The aim is to characterize when the data constrain individual decoder directions and when they constrain only combinations of features.

Stage 2: Test the prediction on controlled models

We will generate synthetic activations with known underlying features:

Independent sparse binary features.

Correlated and frequently co-occurring features.

Continuous circular and spherical manifolds.

Sparse gates controlling continuous manifolds.

We will train SAEs with different seeds and widths.

For each model, we will measure:

Selectivity: How accurately latent firing predicts ground-truth concepts.

Direction recovery: Alignment with known generating directions or subspaces.

Reconstruction: How well the SAE explains held-out activations.

Intervention faithfulness: Whether changing a feature produces the expected change in the underlying variables or downstream computation.

We will construct alternative decoders that preserve held-out reconstruction as closely as possible while changing individual feature directions. Perturbations will be controlled for magnitude and decoder constraints.

The main test is whether correlations and near-dependencies between latent activations predict instability in the effects of individual interventions.

We will also compare interventions on individual latents with interventions on the subspaces spanned by groups of related latents.

Stage 3: Test on a small transformer

We will first use a small transformer trained on a controlled task, such as modular arithmetic, where the relevant variables and expected counterfactual behavior are known.

We will then test the most promising findings on one pretrained language model using a public SAE.

Across different SAE seeds or approximately reconstruction-preserving decompositions, we will compare:

Stability of latent firing patterns.

Stability of decoder directions and feature subspaces.

Detection of semantic concepts on held-out examples.

Agreement of downstream effects under matched interventions.

We will evaluate interventions against specific counterfactual predictions, not merely whether they change output logits. We will also measure unintended effects on unrelated outputs.

A central question is whether feature groups remain functionally stable when individual features do not.

Our main hypothesis is that correlated sparse codes can support multiple good reconstructions with different individual decoder directions. We expect the ambiguity to be larger when latents frequently coactivate or tile the same continuous manifold.

This leads to three predictions:

Activation selectivity can remain stable when decoder directions are not. A concept may be reliably detected without having a unique associated intervention direction.

The covariance structure of sparse codes predicts decoder ambiguity. Small eigenvalues should permit larger reconstruction-preserving changes, subject to decoder constraints.

Feature groups can be more stable than individual latents. Subspace-level interventions may agree across decompositions even when individual-latent interventions differ.

These are hypotheses, not assumed outcomes.

A negative result would also be useful. If real SAE dictionaries remain tightly constrained despite correlated codes, we want to identify which properties of their training or activation distributions resolve the ambiguity.

What Would Count as a Strong Result?

The strongest outcome would combine:

A mathematical characterization of when sparse-code statistics fail to identify individual decoder directions.

An empirical demonstration that the predicted ambiguity appears in trained SAEs, including on held-out data.

Evidence that this ambiguity matters for causal interpretation, or that an alternative representation, such as a feature subspace, resolves it.

We will distinguish changes caused by genuinely ambiguous decompositions from artifacts of off-distribution interventions. In particular, an arbitrary single-latent intervention may not correspond to a valid change in the underlying concept.

The goal is not to claim that individual SAE features are generally meaningless. It is to identify the conditions under which their interpretation is justified.

Within three months: Derive the basic identifiability result, test it on synthetic features and manifolds, and perform an initial causal evaluation on one small transformer.

Within a year: Extend the analysis to pretrained language models, develop conditions for stable recovery of sparse concept-presence variables, and test whether feature-group interventions are more robust across SAE decompositions.

Dropped first if progress is slow: Multi-model comparisons, new SAE architectures, and large-scale autointerpretability evaluations.

The minimum successful project is one clear theoretical prediction and a controlled experiment that tests it.

We’ve learned about transformers learning to represent belief-states in their residual stream. We also learned that certain concepts inherently live on submanifolds of the residual stream (such as the days of the week, or the months of the year). How might these two understandings of activation geometry interact in a toy model that implicates both perspectives? In particular, are concept manifolds the low-dimensional projections of the belief-states of a continuous latent variable?

I can think of a few reasons it matters:

• 1)Unifying two perspectives – on the one hand, the manifold geometry results tell us that certain concepts live on submanifolds of the activation space. On the other hand, toy transformers learn to represent beliefs about concepts.
• 2)Allowing for uncertainty – belief states encode uncertainty about latent variables, while concept manifolds don’t really incorporate this.

• 1)Current research on belief-state geometry (papers [1] [2] [3]) only uses hidden Markov models with discrete states.
• 2)Research on concept manifolds (papers [4] [5] [6]) mostly refer to exact concept manifolds, rather than uncertain beliefs about latents underlying those concepts. For example, the day-of-the-week manifold and the line-count manifold don’t necessarily encode information about uncertainty in these variables.
• 3)Karkada et al. [7] derive the geometry of concept vectors from certain symmetries about the pairwise co-occurence of their corresponding tokens in the pretraining corpus. Claude claims the machinery in this paper can be interpreted as a continuous-state HMM with no transitions (i.e. always constant), leaving open the question of updating beliefs
• 4)Other work ([8] and [10]), while giving arguments for why these representations are learned, still do not have the perspective of belief-state geometry. I need to read these papers in more detail though
• 5)According to Claude (citing papers [11][12][13]), the case of a continuous-state HMM which emits tokens from a finite, discrete alphabet is not in general reducible to a finite-state HMM, since the former has infinite Hankel rank while the latter has finite.

The core steps behind the project are the following:

• 1)Design a toy hidden Markov model that has a continuous latent space and produces strings of letters. An example could be a model whose latent variable drifts along a circle (something corresponding to time of year)
• 2)Train a small transformer on a corpus of strings generated by this toy model. Hopefully, if it predicts the next token well, it is learning a good representation of the underlying latent space
• 3)Do some statistical analyses of the learned activation geometry inside the model. We could learn a linear probe to read out the model’s belief about the latent variable in terms of the activation. Another option is PCA (following Shai et. al [1]) to get a view of what the learned concept manifold/belief geometry is. Check how this compares to the exact theoretical predictions for an optimal Bayesian learner for the HMM.
• 4)Rule out any boring explanations, perhaps by doing some causal interventions to ensure that the transformer’s representing the belief of the latent is necessary for the prediction it makes. If we can’t, we go back to step 1.
• 5)See if we can make any connections to actual models (such as Gemma 2b or 9b). One interesting image is Figure 14 from Karkada et al. [7] in Appendix C. The fact that “may” is off the concept-manifold is due to ambiguity in meaning (either the month or the verb). Could this be an analogue of uncertain states living in the interior of the belief simplex?

Karkada et al. [7] do some Fourier analysis to obtain the shape of the concept manifold in higher dimensions (so the first two principal components are a circle, while higher order components are more complicated lissajous curves). Since the belief states about a continuous latent variable naturally live on an infinite-dimensional simplex, hopefully the transformer activation geometry learns a low-rank projection of this simplex (perhaps by keeping the largest components of the Fourier transform).

For the circle example, some concrete predictions could be:

• 1)The first two principal components form a disk, not just a circle, and the interior of the disk corresponds to uncertainty in the true state of the latent
• 2)It keeps only a few Fourier components.

I would be excited if I did PCA on the activation stream and saw some circle or some other recognizable geometry related to the structure of the HMM (as previously seen with belief-state geometry).

I would be disappointed if simpler explanations were available for the transformer predictions such as counts of recent letters and there was no indication of belief geometry.

The core plan (steps 1 - 4) could be done in a week, but it might require multiple iterations to get an interesting continuous HMM. After that, studying the particular geometry of the model and applying the theory to Gemma might take a month. I would drop the connection to Gemma first if it turned out to be inapplicable.

Claude says the project will require about 100 GPU-hours.

Can Singular Learning Theory be recovered from the thermodynamic limit of high-dimensional statistical models? In particular, do the thermodynamic and low-temperature limits commute, and can the resulting free energy be described through emergent statistical properties of singular parameter-space geometries?

Singular Learning Theory (SLT) provides a mathematical framework for statistical inference in models with singular parameterizations, including neural networks. The asymptotic regime underlying this theory is, however, formulated at fixed parameter-space dimension.

To place this regime in the broader framework of statistical physics, consider a family of partition functions of the form

where $H_N$ is the Hamiltonian, $\pi_N$ is a reference measure on the parameter space, $N$ denotes the system size, and $\beta$ is the inverse temperature.

Two distinct asymptotic regimes are naturally associated with this expression: the low-temperature limit $\beta\to\infty$ at fixed $N$, and the thermodynamic limit $N\to\infty$ at fixed $\beta$. In the former, the Gibbs measure concentrates near the minima of the Hamiltonian. In the latter, the increasing number of degrees of freedom may give rise to collective phenomena and emergent order parameters.

Although finite systems may already exhibit collective behavior, the thermodynamic limit provides the natural framework for describing macroscopic order and genuine phase transitions. Such phenomena are not directly captured by fixed-dimensional low-temperature asymptotics.

Moreover, the two limits need not commute. Rather than representing a limitation, their non-commutativity may encode different physical properties of the system.

In Bayesian inference, the number of observations $n$ plays the role of an inverse temperature, since the likelihood enters the partition function through the factor $e^{-nL_{n,N}(\theta)}$, where $L_{n,N}$ is the empirical average negative log-likelihood.

From a geometric viewpoint, it is natural to consider instead the population Kullback–Leibler divergence $K_N(\theta)$ and the corresponding partition function

For fixed $N$, the asymptotic behavior of this integral can be studied by Laplace's method. The main difficulty in singular models is that the set of minima may form a possibly singular analytic variety, preventing a direct application of the classical non-degenerate Laplace approximation.

Using methods from algebraic geometry, Watanabe's theory provides, under suitable assumptions, the expansion

where $\lambda_N$ is the real log canonical threshold and $m_N$ is its multiplicity [Watanabe2013].

The expansion is obtained at fixed $N$, and its remainder is not necessarily uniform in the dimension of the parameter space. Consequently, it does not directly determine the thermodynamic limit or justify exchanging the two limiting procedures.

On the other hand, statistical-mechanical approaches to large neural networks already investigate thermodynamic limits and collective order parameters [Pacelli2023, Li2021]. The relation between these descriptions and the geometric invariants appearing in SLT remains to be understood in general.

The purpose of this project is to investigate whether a thermodynamic description of singular statistical models can be obtained in terms of distributions of suitable geometric observables, and whether the subsequent low-temperature limit allows one to recover, or extend, the asymptotic theory of Watanabe.

We propose to investigate the thermodynamic limit of the normalized free energy,

followed by its low-temperature asymptotics as $n\to\infty$.

Our research will proceed along three directions.

First, non-commutativity of limits.

We will construct families of singular statistical models with an increasing number of parameters and investigate whether the classical fixed-dimensional asymptotic expansion remains valid in the thermodynamic regime. This includes identifying suitable energy normalizations and comparing the two orders of limits after removing their leading divergences.

Second, random geometry.

We will investigate whether collective geometric observables, such as distributions of local degeneracies, Hessian ranks, or singularity types, admit well-defined thermodynamic limits. The goal is to understand whether the free energy can be related to the statistical distribution of these observables, rather than exclusively to fixed-dimensional singularity invariants, following the concentrations of order parameters picture in statistical physics.

Third, quantitative predictions.

Rather than fixing a model a priori, we will select it by explicit criteria: (i) exact or numerically controlled free energy at finite $N$; (ii) known or computable RLCT at each $N$, so that the fixed-dimensional prediction exists; (iii) a natural scaling of $K_N$ and $\pi_N$ with $N$ yielding a non-trivial limit; (iv) coupling between parameters, since factorizing models make the limits commute trivially; (v) measurable geometric observables. We will start from a non-interacting control satisfying (i)–(iii), then add coupling, and compare finite-size free energies against classical SLT and the thermodynamic description.

Importantly, we do not aim to prove a general theorem for arbitrary neural networks within three months. Even one explicit model exhibiting non-commutativity, or a demonstrable improvement in finite-size predictions, would constitute a meaningful first result.

Our main hypothesis is that the thermodynamic and low-temperature limits do not commute in certain classes of singular models. In these cases, the large-dimensional free energy may depend on collective geometric quantities that are not captured by the fixed-dimensional RLCT and its multiplicity alone.

We expect that a statistical description of singular geometries could provide more accurate predictions when both the model dimension and the number of observations are large, particularly outside the regime where classical asymptotic expansions are uniform.

A more ambitious possibility is that Watanabe's invariants can be recovered as low-temperature characteristics of an emergent random geometry.

However, commutativity would also be informative. If the limits commute in our examples, we will investigate the structural conditions responsible for this behavior and determine the regime in which standard SLT predictions remain reliable.

The immediate objective is a mathematical investigation of tractable singular models, rather than a complete theory of deep neural networks.

Within three months, we aim to produce at least one explicitly analyzable model, a rigorous or quantitatively controlled comparison of the two limiting procedures, and numerical results testing the accuracy of thermodynamic versus fixed-dimensional predictions.

If the random-geometric formulation proves tractable, we will also attempt to identify its relevant collective observables.

The longer-term goal is to extend this framework to neural networks, connecting singular learning theory with the statistical mechanics of large-scale learning systems.

Recent work by the Simplex has shown belief geometry emerging gradually [1, 2], with deeper details of the fractal appearing at different training times. How does the effective model complexity, as measured by the Local Learning Coefficient, change as the model learns richer detail of the belief-state geometry?

To know whether our models generalise beyond training, we want to know whether they build a representation of the world beyond the tokens.

Shai et al. showed that transformers trained on the tokens of a hidden Markov model build approximations to the belief-state geometry (BSG) of an ideal predictor: the transformer builds a world model in order to predict. We can only see this because we know the ideal BSG in the first place.

Singular learning theory (SLT) links generalisation to effective model complexity, which the Local Learning Coefficient (LLC) estimates: at equal training loss, a lower LLC means a lower Bayesian generalisation error.

Relating the growth of the BSG to the LLC would test an SLT measure of complexity against a world model whose ground truth we know. Calibrated on known processes, the relation would let us infer when transformers learn the BSG of processes whose hidden Markov model we do not know.

Shai et al. watched the belief-state geometry form over training and, in a companion post, asked whether its development relates to phenomena in singular learning theory [1]; the question is still open. In October 2026, Gietelink Oldenziel, a co-author of [1], described the two programmes as largely orthogonal: SLT studies training in weight space, belief geometry studies inference in activation space [12]. Studies of belief geometry over training describe its formation without a measure of complexity. For factored processes, the activations compress, rearrange and plateau in a "representational attractor" [2]; over 100M steps on Mess3, the probe error falls as a power law, then dips [3].

The LLC can be estimated by sampling in networks of this size [4]. In small transformers trained on language and on in-context regression, critical points of the LLC curve, where its rate of change vanishes, mark developmental stages that coincide with new internal structure, such as n-gram statistics and induction heads; the LLC rises in some stages and falls in others [5]. Refined LLCs follow this development head by head [6]. In a toy model of superposition, SGD passes through critical points of increasing LLC [7]. In grokking, the network moves from a higher-LLC memorising basin to a lower-LLC generalising one [8].

None of these studies trains on a process whose belief geometry is known, so none can set the LLC against what the network has learned. The geometry has its own computable complexity: the memory needed to track the beliefs diverges at a rate set by the statistical complexity dimension [9].

First, we would validate the depth measure. Depth-k beliefs, the Bayesian filter on the last k tokens, locate the belief to a level-k cell of the geometry. The cells shrink exponentially, so the plain R² saturates: on Mess3, a representation that encodes only the depth-4 belief already scores R² = 0.98 against the full belief. We would therefore score the increment each level adds, normalised by its variance, on held-out prefixes. Synthetic representations encoding exactly depth K must return K.

Then we would train the paper's transformer on Mess3 and RRXOR, several seeds each, and set the LLC [4] against the resolved depth at log-spaced checkpoints, as many as a pilot of cost and precision allows.

We would place stage boundaries at critical points of the LLC curve [5] and ask four questions:

• 1.Timing: does each critical point mark a new resolved level of the belief state geometry?
• 2.Location: which layers and heads store the new detail (refined LLCs [6]), and is RRXOR's complexity spread across layers like its geometry, which was found only in the concatenation of all layers in [1]?
• 3.Scale: does the final LLC grow with the geometry's statistical complexity, across Mess3-family processes of differing statistical complexity dimension [9] and RRXOR, whose belief set is finite? Two independent Mess3 copies test factoring [2], which PCA can hide; we conjecture their LLC approaches the sum of the factors'.
• 4.Regime: does the LLC track the geometry or only the loss? In our preliminary results, lazy networks (trained with Chizat–Bach output scaling [10]) separate loss from geometry: they predict well while their geometry stays at its initial level. We would report their representation's movement, the probe's gain over initialisation and the excess loss, and calibrate the sampler per α and process, since output scaling multiplies the loss curvature by α².

If compute allows, we would predict each process's resolved depth from its LLC alone, with the relation fit on other processes.

We test three hypotheses.

Association: over training, the LLC tracks the resolved depth of the geometry beyond what the loss detects.

Staircase: each critical point of the LLC coincides with one new level of the fractal, where the loss reaches that of the order-k Markov predictor, by analogy with the toy model of superposition, in which SGD passes through k-gons of increasing LLC [7]; compressions such as the one that precedes the factored representation [2] would appear as falls.

Scale: across processes, the final LLC grows with the statistical complexity dimension. This one is a conjecture: Marzen and Crutchfield tie the number of features near-optimal prediction needs to the fractal dimensions of the belief distribution [11], but no result yet orders network LLCs by these resources.

We would drop the staircase if critical points and new levels do not coincide, and the scale hypothesis if the final LLC tracks the loss or the entropy rate rather than the complexity of the geometry. We would drop the question if the LLC does not track the resolved depth beyond what the loss explains, or if lazy networks, which in our preliminary results keep the geometry at its initial level, trace the same LLC curve as rich ones.

We aim to produce LLC curves over training for Mess3 and RRXOR, and possibly for a factored process, set against the resolved depth of the belief geometry, together with a stage-by-stage map between the two. The level of detail, the number of process variants, seeds and network sizes, and extensions such as refined LLCs per head and the held-out prediction depend on funding. The models are small; the cost lies in estimating the LLC at many checkpoints, with the devinterp library.

Let $R(w) = \frac{\gamma}{2}\|w - w^*\|^2$ be the localization penalty, anchored at a moving point $w^*$. Lin's conjecture: If you do deep learning with this regularization then you are doing internal model selection. For which γ is this true, and what happens otherwise?

A network can solve a task in several ways, each a region of weight space with its own loss $L_\alpha$ and learning coefficient $\lambda_\alpha$, the complexity measure of singular learning theory (SLT). Bayes picks the region with the smallest free energy $nL_\alpha + \lambda_\alpha \log n$, switching to a more complex region as data grows. Those switches are the phase transitions and stages that developmental interpretability reports, using the local learning coefficient (LLC), which is estimated with exactly the penalty R. So R both measures complexity and, if Lin is right, makes training choose regions by free energy. If true, trained networks perform Bayesian model selection internally. If it fails near transitions, where stage claims are made, they are artifacts.

Definitions. Fix a region Ω containing $w^*$, let $w_0 = \arg\min_\Omega L$ and $r = \|w^* - w_0\|$. Let λ(w₀) be the learning coefficient (RLCT of K − K(w₀)) at w₀. Let $F_\gamma(w^*) = -\log \int \exp(-n\beta L_n(w) - R(w))\, \varphi(w)\, dw$ be the localized free energy, and define the finite-n estimate $\hat\lambda(n,\gamma) = n\beta(E_p[L_n] - \min_\Omega L_n)$, the quantity the LLC estimator computes.

Open problem (Lau et al., arXiv 2308.12108, App. D). They write: "we conjecture that $\tilde\lambda(w^*)$ is equal to the LLC $\lambda(w^*)$ given certain conditions on γ … ascertaining the exact conditions on γ [is] left as future theoretical work." Current practice is the heuristic "keep γ between 1 and 10".

A. Prove restricted localization results with explicit errors

B. Use a fully specified polynomial benchmark

Retain

For evidence calculations, take $W=[-1,2]\times[-1,1]$, uniform prior density $1/6$, and fixed regions $U_0=W\cap\{u<1/2\}$ and $U_1=W\cap\{u>1/2\}$.

C.Fall back: Prove the theorem for monomial losses or separable losses (where the integrals factorize into one-dimensional ones).

Estimator window: widest at the crossing, closing past it; selection window: the reverse. Toy: gradient descent splits by basin volume, independent of λ; Langevin and anchored training shift toward the quartic well by the Bayesian gap 0.25 log(nβ). Plain SGD switches later than Bayes.

What would change our mind, and what it would mean. (i) Anchored training matches gradient descent's split under every γ schedule: descent on F_γ never crosses regions, so R measures but does not select, and the conjecture reduces to the known estimator. (ii) Its preference is flat in n or favours the regular well: selection is driven by the O(1) term (extent of the low-loss set), not λ log n, so LLC "stages" track valley geometry, not complexity. (iii) Plain SGD matches Bayes at every learning rate and batch size: degenerate noise is no obstacle, R is idle, and the strong conjecture holds.

Three months: the γ-window theorem with explicit error terms; a γ-selection rule contributed to devinterp; the polynomial and reduced-rank benchmarks comparing anchored training, Langevin and SGD against Bayesian region occupancy; a short paper resolving Lau et al.'s conjecture and stating exactly when "deep learning with the localization penalty is internal model selection".

In a tanh network at an exact realisation with one sextic transverse direction, there is a declared range of inverse temperature and localisation, within a stated budget, where the exact localised coefficient exceeds the Hessian prediction by a persistent 1/6 absent in a regular nonlinear control, and SGLD resolves it.

Terms (not counted). Three quantities are kept apart: λ, the asymptotic learning coefficient (Watanabe's real log canonical threshold [7]); $\Lambda(t, \gamma) = -t\, \partial_t \log Z(t, \gamma) = t\, E[K]$ under the posterior $\propto \exp(-tK - \gamma\|w - w^*\|^2/2)$ on a declared domain, the exact finite-temperature localised observable; and $\hat\Lambda$, what the sampler returns. The Hessian prediction is $G(t, \gamma) = \tfrac{1}{2} \sum_i t h_i/(t h_i + \gamma)$ over the Hessian eigenvalues $h_i$, exact for a quadratic loss. The measured excess decomposes as $\hat\Lambda - G = (\lambda - r/2) + [(\Lambda - \lambda) - (G - r/2)] + (\hat\Lambda - \Lambda)$: the asymptotic geometric excess, the finite-temperature and localisation correction, and the sampling error. "Persistent" means the first term, the part that does not decay as t grows. At smooth points of the zero-loss set it is Kurumadani's (c − r)/(2k) under his nondegeneracy conditions [1, 2], with r the Fisher rank, c the codimension and k the transverse jet order; the sextic case is k = 3, excess 1/6.

Developmental interpretability reads local learning coefficient estimates along training as effective parameter counts, and Timaeus's own guide says the sampler's temperature, localisation and step size have no defaults and must be swept [6]. At smooth points of the zero-loss set, what the coefficient adds to curvature is a definite number with a definite cause, (c − r)/(2k). But a finite-temperature excess over the Hessian prediction is not evidence of it: the regular loss tanh²w gives Λ − G = 1/(2t), a fixed Gaussian localiser lets a distant flatter zero take over as t grows, and a nearly singular εv² term imitates the singular case for γ² ≪ t ≪ ε⁻². Whether a practical measurement can see higher-order geometry at all, in which range and at what cost, is what anyone reading an LLC curve needs to know. A negative result names the constraint that blocks detection.

Kurumadani [1] proves λ = r/2 + (c − r)/(2k) at smooth points of the realisable set under jet nondegeneracy, and his dissertation [2] computes the two-unit tanh case with zero input weights explicitly, (d₁, r, m) = (2, 1, 3), λ = 2/3 under a₁² ≠ a₂², with the four-unit extension; the asymptotic answer for the test case is settled, which is what makes it usable as ground truth. Wang et al. [3], Appendix D.3, state λ ≥ r/2 and find Hessian-rank estimates unreliable. Cai [4] derives continuous-time Gaussian-localised targets for u² and u²v² (a Bessel-function closed form for the centred product singularity), separates them from finite-step Langevin effects where a closed form exists, and gives the mean energy under a Gaussian surrogate (his Proposition 1), all in service of a thermodynamic-variance estimator on MLP trajectories; he has no regular nonlinear control, no smooth-fibre sextic geometry and no network prediction. Hitchcock and Hoogland [5] benchmark sampler fidelity to the local posterior. Nobody has written down a diagnostic that separates a persistent excess from the O(1/t) correction of a regular nonlinear minimum, derived the visibility scale $t \asymp \gamma^k/a$ with the coefficient and localisation metric taken in a network's own coordinates, or tested whether an actual tanh network's 1/6 can be resolved within a budget. That is the gap.

First, the benchmark. Four losses with the same Hessian diag(2, 0) at the origin: u² with an unused v (λ = 1/2, Gaussian calibration); tanh²u with an unused v (λ = 1/2, the regular nonlinear control, Λ − G = 1/(2t)); u² + a v⁴ (λ = 3/4, persistent excess 1/4); u² + b v⁶ (λ = 2/3, excess 1/6). Kept separate: $K_\varepsilon = u^2 + \varepsilon v^2 + v^4$, whose quartic regime γ² ≪ t ≪ ε⁻² exists only when εγ ≪ 1, as a resolution test; and u²v², with its 1/2 − 1/log ρ approach, as a slow-convergence test. For each, compute Λ(t, γ) by quadrature on a declared domain, localiser imposed in the stated coordinates, and G on the same domain.

Second, the diagnostic, fixed before any network result. Over a declared t-range, fit Λ − G to A + B/t + C/log t; the excess is persistent when A separates from zero at the stated resolution while the tanh² control returns A = 0. Quantify the minimum detectable curvature $\varepsilon_{\min}$ from the $K_\varepsilon$ window. Then run SGLD at the same (t, γ), varying step size and chain length separately, to measure $\hat\Lambda - \Lambda$ and the cost of resolving 1/6.

Third, the network. Two students with zero truth: the two-unit network at (a₁, a₂, b₁, b₂) = (1, 2, 0, 0), and a four-unit student made of a fixed nonzero two-unit teacher plus a zero pair, (r, c, k) = (5, 6, 3), λ = 8/3, teacher units free. Evaluate the actual loss in the original coordinates with the localiser imposed there; derive the sextic coefficient (1/700 before KL normalisation) and the induced localisation metric; predict the visibility range t ≍ γ³/a and its cost; then test the prediction with the diagnostic of step two.

Fourth, deferred until the first three work: one training trajectory with the checkpoint-generated divergence $D(p_{w_t} \,\|\, p_w)$ as the loss, stated as such.

Prediction. On u² and tanh² the diagnostic returns A = 0, with B = 1/2 for tanh²; on the quartic and sextic controls it returns A = 1/4 and 1/6 once $t \gg \gamma^k/a$; $K_\varepsilon$ returns A = 1/4 inside its window and 0 outside; u²v² approaches 1/2 slowly enough that a short t-range misreads it. In the two-unit network the predicted visibility range is nonempty at laptop cost and SGLD returns A = 1/6 within the stated resolution; the four-unit student does the same at higher cost.

What would make me drop it. If the predicted visibility range is empty within the budget, or the diagnostic cannot separate the sextic control from tanh² over any affordable t-range, then higher-order geometry at this order is not measurable with this workflow in this family. The report then names which constraint blocks detection, locality, the finite-temperature correction or the sampler, and how it scales with k and the coefficient. That is a result, not a failure, and it licenses no claim about published LLC curves in other models.

Two weeks give the benchmark, the diagnostic with its false-positive check, and the two-unit network prediction with one test of it. That is the decision point. Three months, only if the two-week criterion is met: the four-unit student, the visibility-and-cost result written as a short paper, and a second geometry (a coinciding pair, k = 2) for the scaling in k. Needs a laptop, the quadrature code from the first proposal's pilot, and devinterp. Drop first the four-unit student, then $K_\varepsilon$. Postponed as separate problems: trajectories, the hidden-noise limit of the layer-composition bound, and complex thresholds of fibre ideals.

Can certifying that a monitor can match an LLM’s use of its written reasoning require exponentially many tests, even in a simplified model where each test measures both systems’ best performance exactly and their abilities without the reasoning match?

Suppose an LLM writes reasoning that helps it solve a problem. A trusted monitor reads the same text and performs equally well across many evaluations. Can we conclude that the LLM has no important use for that reasoning the monitor misses?

If ruling out this advantage requires exponentially many tests, accurate evaluations alone cannot provide a general guarantee. We would need additional assumptions about how the models use their reasoning, or access beyond evaluation scores.

If a small, systematically chosen set of tests suffices, it could give auditors a justified stopping rule and guarantees covering untested objectives. Either outcome would clarify what passing evaluations actually establishes. Applying either conclusion to real LLMs requires checking that they fit the mathematical assumptions.

Anwar et al. [1] propose detecting hidden information by measuring how much a reasoning trace improves an LLM’s performance compared with a trusted monitor’s. They show that a fixed collection of tasks can miss information useful for another task. They do not determine how many tests are needed when the underlying situations stay fixed and the auditor chooses each new test using earlier results.

Blackwell [2] connects the usefulness of information across all decision problems with the ability to reproduce one information source from another. Le Cam [3] develops approximate versions of this comparison. These results explain why checking every relevant objective can establish a meaningful guarantee. They do not directly tell us how many selected tests suffice for agents with restricted abilities.

There is also a geometric connection: each agent’s possible behaviors form a shape, and a test measures how far that shape extends in one direction. Eisenbrand et al. [4] study the difficulty of determining whether one shape fits inside another, under different access and error assumptions.

The first research task is to check whether those existing results already settle our monitoring question. The project is worthwhile only if the monitoring assumptions leave a substantive gap.

First, construct a small example with two equally likely situations and two possible actions. Give the model and monitor the same choices without a reasoning trace, but let the model use the trace to choose its action differently in each situation. Calculate which tests reveal this advantage and which miss it. This gives a concrete starting point for checking the definitions.

Next, use the Blackwell connection to define what successful monitoring should guarantee: for each model strategy, the monitor can reproduce approximately the same pattern of actions across situations. Matching overall action frequencies is insufficient; the actions must occur in the right circumstances.

Then fix the auditor’s access. It chooses scoring rules and receives both systems’ exact best scores, but cannot inspect their internal rules. Its next test can depend on previous answers. Keeping the situations and their probabilities fixed separates test selection from changes in the environment.

Build larger examples in which a substantial advantage is revealed by only a small fraction of possible tests. Try to prove that finding it requires exponentially many tests as the number of situation–action pairs grows. This is not a claim about exponential growth in model size.

Compare the construction with known geometric results before claiming novelty. Finally, look for assumptions that make certification easier, beginning with a model whose available strategies are explicitly listed. These comparisons would identify which restrictions create the difficulty and which additional information removes it.

I expect some model–monitor pairs to agree on many tests while differing substantially on an objective those tests miss. Even choosing tests in response to earlier results may fail to uncover the difference quickly. This is a prediction about the simplified setting, not evidence that real LLMs routinely hide such advantages.

I expect certification to become easier when the auditor knows a short list covering the model’s possible strategies. That would give a concrete example of additional knowledge making evaluation results more informative.

A method that reliably detects every substantial advantage with only polynomially many tests would refute the proposed exponential requirement. I would then investigate that positive result. I would drop the project as a standalone contribution if existing geometry already settles the question and no meaningful monitoring restriction remains unresolved. I would reconsider its LLM relevance if the difficult examples depend entirely on implausible restrictions on the monitor.

Three months would produce a precise model, worked examples, and a proof that many tests are necessary or a method showing that fewer suffice. The first month would check existing results and build examples; the second would develop the main argument; the third would examine one easier case and write the findings.

The work needs a laptop, optimization software, and advice from researchers in decision theory and geometry. It does not initially require large-model experiments. If time becomes tight, I would drop extensions involving noisy scores or changing environments and keep the question about exact tests in a fixed setting.

Can models learn, from realistic RL training setups, to write steganographic notes to themselves? I’m specifically interested in demonstrating that this is feasible.

I think models are clearly able to encode hidden messages in their outputs (cf. classic subliminal learning setup). As we rely more and more on agentic coding, where models leave many notes for themselves (Claude’s briefs, Anthropic’s ‘git for agents’, etc.), it’s crucial for safety that we understand what’s being communicated in those notes. Sufficiently powerful models could also use general steganographic capabilities to pursue other misaligned goals (e.g., exfil), and this would greatly weaken our monitoring defenses. If it can be shown that a reasonably realistic RL setup could lead capable models to develop such capabilities, labs will hopefully be pressured to rethink and tighten their safety measures.

Training against CoT monitors makes models obfuscate their reward hacking, but this happens only within a single context [1]. Penalizing specific strings in the CoT makes models encode them [2]. Steganographic collusion has also been demonstrated [3, 4]. As far as I know, no one has studied whether models can learn, and generalize, to produce benign-looking steganographic text that only the same model can understand, with no shared context.

I would run RL in a carefully designed environment. In particular, each rollout would consist of two phases, with context flushed in between:

Exploration: the agent is given the task context and an opportunity to look around the environment, where a specific reward hack is implanted. It’s also told that it will have to solve the task in a second phase, with a fresh context but a shared scratchpad, and that it can leave notes for itself there.

In the second phase, the model is encouraged to read its scratchpad and conclude its work. Depending on how experiments go, we could remove the hack from the environment in this phase.

The reward includes a penalty term from a weaker monitor model. The monitor has access to the scratchpad but not the CoT. Crucially, we can make the RL task impossible to solve without the hack, so that the model would be pressured to encode it in its scratchpad. Optionally, before RL, I would run SDF on documents describing steganography as a way to evade monitors. I think this setup, while synthetic, is not that far from real-life agentic RL environments.

I expect weaker models to likely fail to learn anything meaningful.

With more capable models, I think the right optimization pressure is there, and I’m genuinely uncertain if they’ll learn.

The nature of the hack is also important. If it’s something syntactic and simple, it could learn a trivial encoding that’s not practical for any actual use cases. It should ideally be something semantically rich.

Hopefully, whatever steganography it learns would be obvious from reading its CoT.

Most interesting / worst-case scenario: it gets the reward with no obvious signs of reward hacking, both in its scratchpad or CoT. Steganography generalized to CoT.

Three months should be plenty provided enough compute. I would probably start with a setup that’s most likely to succeed (a lot of cues, easy to find hack, etc) and see what’s the smallest model that learns anything. One cue could be continued pretraining on steganographic text. If I get a working setup, I would try to find a minimal version and test if these steganographic capabilities generalize beyond the RL task (e.g., to standard agentic coding).

When two AI systems hide information in harmless choices about how they express the same reasoning, we conjecture that, in a simple model of structured rewriting, eliminating covert communication requires changing essentially the entire message.

Language gives us enormous freedom to express the same idea: we can change words, examples, ordering, or the structure of an explanation. Two AI systems could use these harmless choices to communicate information that a monitor does not notice.

A natural fix is paraphrasing: rewrite one model’s reasoning before another model sees it. The basic quantitative question is: how much rewriting is needed to reduce the maximum possible rate of hidden communication?

If a modest amount of paraphrasing can drive this rate to zero, paraphrasing could be a strong general defense. However, there is a natural tension between preserving useful reasoning and preventing covert communication.

Zhou and May [1] show that language models can hide information in high-level reasoning choices rather than particular words, and that these signals can survive ordinary paraphrasing. A paraphraser that knows the particular hidden strategy performs substantially better.

Rozinek [2] comes closest to a mathematical treatment. It separates the intended meaning of a message from the freedom in how that meaning is expressed and studies the information that survives a fixed paraphrasing channel. However, it does not characterize the full tradeoff between how much rewriting is allowed and how much covert communication remains.

We study this tradeoff first in a setting where “same meaning” is exact. Consider different ways of grouping a sequence of statements joined by “and.” For example,

(A and B) and C

and

A and (B and C)

have exactly the same meaning. A local regrouping changes one into the other.

These groupings correspond to binary trees. Sleator, Tarjan and Thurston [3], and later Pournin [4], studied how many local regroupings may be needed to transform one tree into another. Our question is different: what is the smallest set of representative groupings such that every possible grouping can be transformed into one of them within the rewrite budget? Even the best universal paraphraser must leave that many distinguishable outputs, which determines how much hidden information can still be transmitted.

We begin with n statements joined by “and.” Different groupings have exactly the same public meaning, so Alice and Bob can use the choice of grouping to encode a hidden message.

The paraphraser is deterministic. It sees the whole input and may make at most k local regroupings. It must commit to its rule before Alice and Bob choose their hidden code. Alice and Bob know the rule and may choose whichever code survives it best. This models a general-purpose defense rather than one designed against a known code.

We first require zero-error communication: Bob must always recover the hidden message. For deterministic paraphrasers, this gives the same asymptotic rate as allowing the decoding error to vanish.

The problem then becomes a covering problem. Let $K_n(k)$ be the smallest number of representative groupings needed so that every possible grouping is within k local regroupings of one representative. The surviving covert rate is

where δ is the fraction of the structure that the paraphraser may change.

We have two starting bounds. A direct neighborhood-counting argument gives g(δ) ≥ 2 − H₂(δ) − 2δ > 0 for δ < 1/2, where H₂ is binary entropy. Conversely, every grouping of n statements can be transformed into the same fully right-grouped expression in at most n − 2 local regroupings [5, Proposition 4.3], so g(1) = 0. The zero-rate threshold therefore lies between 1/2 and 1. We conjecture that it is 1: even allowing a subexponential number of representative outputs cannot improve the leading constant in the rewrite budget. We would attack this conjecture by bounding neighborhood sizes and constructing small covers.

We would also compute the optimal covering numbers for small n. This should reveal the shape of g(δ), identify unusually easy or difficult groupings, and suggest the right conjecture before attempting the asymptotic proof.

Finally, we would test whether the same qualitative phenomenon appears in real LLM reasoning. One model would solve benign reasoning tasks while encoding a controlled hidden message through choices such as the order of independent steps, choice of example, or choice between valid solution paths. A paraphraser would then rewrite the solution at different strengths, and a second model would attempt to recover the hidden message. Here paraphrasing strength means the number of allowed local edits, reported as a fraction of the steps or sentences changed.

For each paraphrasing strength we would measure two quantities separately: how much hidden information remains recoverable, and how much ordinary task performance is preserved. This gives an empirical analogue of the theoretical rate-versus-rewriting curve.

We expect g(δ) to decrease as the paraphraser is allowed to make more changes, but to remain positive until the rewrite budget is very large. Our stronger prediction is δ* = 1: eliminating every positive covert rate requires (1 − o(1))n local regroupings, which is essentially the entire structure.

If g(δ) = 0 for some δ < 1, so that a budget of δn regroupings already allows $K_n = 2^{o(n)}$ representatives, then the conjecture is false and universal paraphrasing is more powerful than we expect. We would then try to determine the true threshold and the full shape of g(δ).

For the LLM experiment, we expect generic paraphrasing to reduce hidden communication gradually rather than eliminate it immediately. We would reconsider the relevance of the toy model if mild paraphrasing consistently destroys structural hidden signals while leaving task performance essentially unchanged.

In three months, the main theoretical goal is to obtain a new bound on g(δ), or at least substantially narrow the location of the zero-rate threshold. We would combine proofs with exact computation on small instances.

A secondary goal is one controlled LLM experiment measuring hidden communication rate and task performance as paraphrasing becomes stronger. This is meant as a test of whether the theoretical abstraction captures a phenomenon visible in real models, not as a full empirical study.

If time is limited, we would prioritize the theoretical result and small-instance computation, and drop the LLM experiment first.

Can a backdoor be planted in a transformer’s embedding or attention weights so that, after a specified training process, the full weights are computationally indistinguishable from those of an honestly-trained model, while concealing the backdoor in its weights and hiding a secret trigger in its text?

Assessment of a model’s trustworthiness uses access to weights, behavioral evals, and inspection of the inputs. A backdoor (in which an adversary directly manipulates the outputs of a model, without the user’s knowledge) embedded in the trained attention would test the limits of all three: it could preserve performance, but suddenly activate on text that appears statistically ordinary. The classical example of using sparse PCA for the weights to “trigger” the hidden backdoor in ReLU is one version of the backdoor. Our project will ask whether these properties can be proven together under a training process for transformers; this will combine cryptographic indistinguishability, information-theoretic guarantees for text distributions, and the actual structure necessary in attention that identifies what assumptions such a backdoor would need. Negative results are also important: showing that training destroys such a mechanism, or that an efficient auditor can detect it, would identify concrete limits for the “hidden” model's behavior.

Existing results establish different parts of this goal, and use different meanings for “undetectable”. The proposal will be to find hiding guarantees and a functioning backdoor in one trained attention model. [8] gives the original foundational construction of undetectable backdoors, including white-box results for a certain random-feature model. [1] further obtain statistical indistinguishability results for frozen Gaussian first layers and a sufficiently Lipschitz downstream network. Neither works establish how attention computationally acts when the relevant weights are trained, or how they can be activated by “statistically ordinary token strings”. [2] combines hidden backdoors with steganographic activation, with no reference to obfuscation of the model. [3,4] address a related question: can it be computationally hard to discover an input that activates the hidden behavior, which is technically different from being statistically indistinguishable from the weights. A circuit can be hard to activate, while its structure looks recognizable. [6] provides a route that preserves a covertext distribution and does not automatically supply a decoder inside trained attention.

We will investigate whether the hidden-weight mechanism in [1] can be adapted to trained transformer attention. The aim is to modify one or several heads so that a secret trigger introduces a signal into the residual stream that produces a chosen incorrect response.

First, we will specify the architecture, initialization, training data, optimizer, training budget, and text generator. These define the honest baseline. We will compare distributions of released weights, seeking computational indistinguishability and, where possible, stronger total variation bounds.

Next, we will build a small attention model with a visible backdoor. The trigger will redirect attention from task information toward an incorrect answer. This will identify the attention scores and output margins required for activation, without claiming either hiding guarantee.

We will then investigate hidden relations among embedding or attention weights. The Gaussian-layer construction and bi-Lipschitz assumptions in [1] provide starting points, but require new arguments for attention due to softmax coupling across tokens and learned query-key structure. Honest and planted models will undergo identical continued clean training. We will track embeddings, attention projections, and predictions: preserving a planted coefficient does not ensure that its effect survives.

Finally, we will develop an encoder whose valid token strings activate the mechanism while statistically matching honest model output. The initial guarantee will cover one sample, averaged over model and secret randomness.

Experiments will measure clean performance, triggered errors, accidental activation, and attention margins. Detection tests can reject candidates, but their failure cannot prove hiding. Success requires distributional guarantees and surviving behavior in the same construction under consistent training and sampling assumptions.

The strongest outcome is a single construction satisfying all four guarantees under a fully specified training process, with a proof of the weight comparison, text comparison, and surviving behavior. A restricted attention architecture would be a meaningful first setting if its restrictions are explicit and both hiding guarantees concern the same construction. This would help with monitoring, controls, and detection for AI safety as well, and we could pivot to an experimental procedure instead.

Our working hypothesis is that training can preserve a sufficiently separated attention mechanism when clean data exert little gradient in the secret direction. This is a hypothesis to test. The toy shows why weak gradients in that direction alone are insufficient: other parameters can move enough to erase activation.

We would change direction if the candidate admits an efficient weight test, its triggering states cannot be reached by valid tokens, or ordinary clean training removes the behavior in the intended regime. A rigorous result establishing one of these failures for an identified class of constructions would be useful. Failure would imply robustness of transformers to obfuscation in general, at least a weak sense.

Within three months, we aim to produce a precise problem statement, a construction or substantive obstruction for one small attention setting, and reproducible experiments. The first month will establish the baseline and transparent mechanism; the second will investigate weight hiding and training dynamics; the third will address text encoding and consolidate results. Understanding exactly how backdoors can manipulate features in the residual stream at a mechanistic level is not obvious and should be explored.

We need theoretical guidance in cryptography and optimization, plus modest GPU access for small models. If integration exceeds this timeframe, we will prioritize the attention and training theorem, explicitly leaving any unresolved text encoding requirement for subsequent work.

On Python benchmark solutions, does rewriting a model's code into one fixed standard form, or into a random equivalent form, cut the hidden bits a decoder recovers from its logical structure to near zero, while the code passes its tests?

• •Setting. In untrusted monitoring, one model copy writes code and another checks it. A shared hidden signal lets backdoors through; one bit suffices.
• •Gap. Paraphrasing removes signals in wording, not logic: which checks code makes, how loops, guards and edge cases are handled.
• •Two model-free sanitisers:

o Canonicalise: map equivalent programs to one.

o Perturb: pick a random equivalent, with a secret seed. Rewrites can be undone, so this never beats a standard form on information; it covers choices too large or complex to standardise.

• •If it works: a model-free defence that cannot be talked out of its job.
• •If not: we learn which logic carries messages, and where monitors should look.

• •Paraphrasing, in text. Paraphrasing cuts simple hiding schemes to under two bits per answer [1]. Bits in high-level reasoning patterns resist a strong paraphraser [2].
• •In code. Gardner-Challis et al. [3] hid a signal in “for” versus “while”. Renaming and reformatting barely helped on BigCodeBench (safety 6.7% to 13.3%). A model-written summary reached 91.7% but hid details the monitor needed. Their minifier never changes control flow, so the middle ground is untested.
• •Adaptive attackers. Learned collusion still emerged under paraphrasing, and a code paraphraser let one bit through over 99% of the time [4]. Randomised defences elsewhere fell once attackers averaged over the randomness [8]; code watermarks with error correction survive random edits [9], so rewrites must reach the program's structure.
• •Neighbouring fields. Network "active wardens" reset every free packet field to a default [10]. Plagiarism detectors undo statement reordering, but not against hidden signals [11].
• •Theory. Rice's theorem rules out a perfect standard form for every program, so the work is finding forms that remove the most hiding capacity.
• •What is missing. We found no work that rewrites code's logical structure without a language model, or measures which hiding channels survive.

• 1.Benchmarks and encoder.
• •BigCodeBench [5] and APPS [6], as in [3], so results compare. HumanEval+ and MBPP+ [7], whose larger test suites leave fewer untested inputs.
• •A frontier model hides bits using a codebook over four channels: names; structure (loop form, guard style, statement order); algorithm choice; edge-case behaviour. A decoder with the codebook reads them back.
• 2.Build the sanitiser.
• •LibCST rewrites at a catalogue of choice sites, each with a safety condition; egglog [12] for typed pure expressions.
• •One switch picks the canonical or a seeded random choice at each site. The rewritten program is the one that runs.
• •Every output passes the tests and differential fuzzing [13]. Code we cannot handle goes to the trusted monitor, and that refusal counts as a possible signal.
• 3.Measure. Each defence is a channel from hidden message to rewritten code. A decoder with bit error rate e gives 1 − H(e) bits per hidden bit, a lower bound, so we also train a classifier as a stronger decoder. Defences: none, renaming and reformatting [3], language-model paraphrasing [1], canonical, random.
• 4.Adapt. The encoder is told the defence and uses error-correcting codes. Channels that keep positive capacity are the main result; we test how much stronger tests close them.
• 5.Control check. Is what survives enough to collude? The “for”/“while” game of [3] on APPS, and Veron's trained colluding monitor [14]: signalled backdoors caught at a fixed false-alarm rate, and the cost to honest detection.

Expect:

• •Canonicalisation drives structural channels (loop form, guard style, statement order) to near zero.
• •Random perturbation leaves a small capacity that error-correcting codes exploit.
• •Both keep pass rates above 95%.
• •Algorithm choice, such as recursion versus iteration, partly survives both.
• •Edge-case behaviour and tie-breaking survive every behaviour-preserving defence, carry most of what remains, and shrink as tests get stronger.
• •In the control check, at most one bit survives, and honest detection stays within five points of no defence.

Would drop the question if:

• •language-model paraphrasing already drives structural capacity to near zero, so a model-free sanitiser adds little;
• •if canonicalisation cannot keep pass rates above 90%, or is often intractable, we focus on the random perturbation approach instead;

Three months produce:

• •a set of program pairs that differ in only one hiding channel, such as loop style, algorithm choice or edge-case handling;
• •an open-source tool that rewrites Python into a fixed standard form or a random equivalent one, checking every rewrite with tests and fuzzing;
• •the capacity of each channel under each defence, in bits per solution;
• •the control-check result.

Needs: API credit for the encoder and paraphraser, a few GPU-hours for the classifier, and a mentor in AI control or programming languages.

Drop first: the HumanEval+/MBPP+ test-strength study, then BigCodeBench.

The models of the same family can sometimes “read each other's thoughts”. We want to study the extent to which models can leverage this acausal coordination (tacit collusion), to transmit complex messages using steganography.

This is important because we can understand how two copies of the same model could conjure up the same steganographic key, thus bypassing the need to transmit it. The key in the steganographic sense can be understood via key-picking and encoding procedure, which can implicitly be represented in the model so that the models can encode/decode messages "subconsciously".

Beyond its relevance to backdooring, this is important for our conceptual modeling of AI agents. For example, applicability of evidential decision theory to agent interactions depends on their ability to communicate tacitly.

There is previous work that attempts to answer parts of this. For instance:

• •In [1], this was done by asking models to play coordination games, such as picking a number from 1 to 100. However, since the models have the same prompts, models of the same type would immediately have the same output logit distribution, so copies agreeing is guaranteed rather than evidence of a shared private key.
• •[2] has a Sender write descriptions of two words that secretly mark one as the target, and an isolated Receiver pick the target. Same-model pairs beat cross-model pairs. However, the descriptions are a message, so this tests decoding a signal.
• •[3] names acausal Schelling-point collusion as a threat to untrusted monitoring and bounds passive self-recognition by fine-tuning GPT-4.1. However, it only studies one model.
• •[4] had two copies of Claude 3.7 Sonnet write a policy and a monitor for a coding task, and found rare collusion on "magic numbers" such as 42 and 1337. Similar to [1], the two prompts were identical apart from the role line.

We start by developing a benchmark that measures models’ abilities to collude/share secret messages across models of the same type and different types. This is done as proof of concept today as well as eyeing the future where the models are more capable.

Next, we will use interpretability techniques to identify mechanistic traces of such self-collisions and intervene accordingly to reduce collusion abilities. This is done perhaps by first looking for “shared decoding key” direction in the activation space, or potentially in the weights itself and see if steering/weight-based ablation techniques would work to lower the collusion abilities.

We believe that models of the same type will have higher ability to collude (i.e. be able to decode secret messages better). This is because we have seen success in “similar architecture” interpretability techniques, such as subliminal learning [5], activation oracles [6], and training models to introspect [7].

We also believe that there should be a mechanistic way to describe and understand the shared key direction/subspace in the weights of the models themselves. This is because if the empirical results (from the first paragraph) is true, combined with the fact that models of the same type have the same architecture, then there might be some sort of mechanistically “shared structure” that helps to explain this phenomenon of tacit collusion/acausal coordination.

If either of these steps fail (i.e lack of ability of collude/interpretability methods does not work as expected), we will then rephrase the question to see why this fails.

• 1.An evaluation benchmark.
• 2.Formalization of what it means to be able to tacitly collude/send secret messages. (This can be done through a behavioral definition, potentially validated through interpretability lens.)
• 3.We need some compute cost to run multiple experiments (ie API/running interpretability on GPUs).
• 4.Mentors with shared interest to bounce ideas off.

Can a weaker LLM, equipped with counterfactual probing and structured decoding, recover hidden messages generated by a stronger adversarial LLM at least 15% points more accurately than a compute-matched baseline, even when the encoding strategy was not seen during training?

CoT monitoring is used to detect unsafe behavior. However, a sufficiently capable model might learn to hide information in its reasoning, making it difficult for a weaker monitor to understand.

Our goal is to investigate whether we can compensate for differences in model capability using better algorithms, not larger models. If a weaker model can recover hidden information through structured probing, we could make monitoring more reliable without requiring the monitor to be as capable as who it supervises.

If this fails, it would suggest that some forms of encoded reasoning create a capability gap that additional inference computation cannot overcome.

Understanding this limitation would help determine when weaker models are sufficient and when stronger monitors/alternative defenses are necessary.

• [1]demonstrated that LLMs can learn encoded reasoning and investigated paraphrasing as defense. Their work focuses on preventing hidden communication rather than recovering the information being communicated.
• [2]introduce the steganographic gap, measuring differences in usable information between a Receiver and a Sentinel. They show that additional monitoring computation can reduce this gap, but do not develop an algorithm for discovering previously unknown encoding schemes.
• [3]investigate the steganographic capabilities of frontier LLMs. They find that models struggle to hide information under standard conditions, but can succeed with additional affordances, including private reasoning and coordination on encoding strategies.
• [4]study how modifying CoT changes model behavior. This establishes counterfactual intervention as a technique for understanding reasoning, but they do not apply it to decoding messages.

More recently, [5] show that steganographic models can evade activation-based detectors and that targeted changes to the evaluation distribution can restore detectability. Their work suggests interventions can reveal hidden information, but does not study adaptive black-box decoding.

Cryptographic constructions [6] also show that keyless detection cannot work universally. Therefore, we focus on learned, noncryptographic encoding strategies.

The open question is whether counterfactual queries can reveal enough structure for a weaker model to recover hidden information.

We would first construct a controlled experiment where a stronger LLM is instructed to hide an 8-bit message inside an apparently ordinary reasoning trace. An intended Receiver, which knows the encoding scheme, must successfully recover the message, while a weaker monitor does not know the scheme.

We would start with Qwen3-32B as the encoder and Qwen3-8B as the monitor, using lexical, syntactic, and semantic encoding strategies. The encoder would generate multiple candidates and select those that evade a baseline monitor while remaining decodable.

Our main intervention would be counterfactual probing. Instead of asking the weaker model to decode the message directly, we would let it modify parts of the reasoning trace and observe how those modifications change the Receiver's downstream behavior.

For example, the monitor could paraphrase half of the sentences and test whether the Receiver still benefits from the trace. If the benefit disappears, the modified sentences may contain information essential to decoding. Repeating this process could help localize the hidden information.

The monitor would then use an LLM to propose candidate decoding rules and small programs, which it could test against calibration examples with known hidden messages. The Receiver-query interface would expose only constrained downstream behavior, not a direct request to reveal the secret.

We would compare this algorithm with direct prompting, few-shot prompting, and additional reasoning tokens, using matched computational budgets. Evaluation would measure exact secret recovery, false-positive rates, and oracle-query complexity on previously unseen encoding strategies.

Finally, we would study a restricted theoretical model where only k of n trace segments carry essential information. Under a reliable, monotone group-testing oracle, these segments can be localized in $O(k\log n)$ queries. We would test when real LLM encodings approximately satisfy these assumptions and when distributed encodings violate them.

We expect that a naive weaker LLM will struggle to decode hidden messages generated by a stronger model, especially when the encoding strategy is unfamiliar. However, we predict that counterfactual probing will improve recovery because the monitor can learn which parts of the text affect behavior.

We expect this advantage to be largest for compositional encoding strategies, where hidden information is distributed across several sentences but is patterned. We expect that structured probing will outperform simply giving the monitor more reasoning tokens.

Our minds change if the improvement disappears when controlling for computation, example access, or encoding strategy. In particular, if counterfactual probing consistently fails on held-out schemes while direct prompting or ordinary fine-tuning succeeds, the proposed intervention would not offer a meaningful advantage.

A second negative result would be finding that the hidden information is too distributed for localization to help, even when the intended Receiver decodes it reliably.

In three months, we aim to produce a reproducible benchmark of adversarially generated hidden reasoning, implement counterfactual probing and structured decoding, and evaluate their performance against compute-matched baselines.

The main deliverable would be an empirical study measuring how secret recovery depends on model capability, encoding complexity, and the number of counterfactual queries. We would also aim for a small theoretical result characterizing when group-testing-style localization is possible.

We would need access to two open-weight LLMs, inference compute for adversarial data generation, and resources for evaluating many interventions.

If necessary, we would drop fine-tuning and focus on prompt-generated-encodings and black-box decoding.

Does cumulant propagation retain its, accuracy scaling with width, when Gaussian-initialized networks receive low-rank weight perturbations?

Wu et al. [1] give an alternative to naive Monte Carlo sampling called cumulant propagation. My aim is to extend the applicability of their algorithm to more realistic training scenarios. Cumulant propagation estimates a network’s expected output by tracking distributional statistics through its layers. It can outperform sampling asymptotically for Gaussian random networks. Useful networks, however, acquire structure through training, precisely where the current guarantee stops applying. We want to know whether cumulant propagation remains accurate after we change a randomly initialized network’s weights. We start with changes chosen independently of the initial weights because these are easier to study than changes produced by training. If the method still works, we can then test whether it also works when training chooses the changes. If it fails, understanding which changes cause the failure could help us improve the method or explain when it can be trusted.

Either outcome would help us move from understanding random networks toward understanding trained networks.

Wu et al. [1] prove mean-estimation guarantees averaged over independent Gaussian weights, for fixed depth and polynomial activations. They also report ReLU experiments and explicitly suggest decomposing weights into structured and random components. They do not establish the corresponding error bound. LoRA [2] motivates studying low-rank updates, but its pretrained base differs substantially from our Gaussian base.

Consider width-n, zero-bias MLPs with standard Gaussian inputs and fixed activation polynomials. Write

The candidate claim is

Here $L$ counts linear layers; rank, norm, depth, and activation polynomials stay fixed as width grows. Each network’s input expectation is estimated first; estimation error is then averaged over Gaussian initializations.

If we choose a fixed weight change before seeing the random initialization, we are just adding fixed numbers to independently sampled weights. The weights remain independent, although their average values change.

Training is different: it chooses changes based on the network’s current weights and behavior. Those changes can make different weights depend on one another.

Our experiments test only the first situation, using ReLU networks and one particular family of fixed changes. The results are encouraging, but they do not yet show that the method works after training. This is a motivating future direction to study.

We tested 1,792 networks at seven widths between 8 and 64. At each width, we compared 128 random initializations with and without the same small low-rank weight change. All networks used ReLU activations and three linear layers.

The estimation error decreased with width at approximately the expected rate. Doubling the width reduced mean squared error by roughly four times for order-2 cumulant propagation and eight times for order-3. At width 64, adding the weight change increased average error by less than 1% for both orders. At width 8, the order-3 error increased by about 14%, suggesting that narrower networks can be more sensitive.

We used repeated sampling to estimate each network’s true expected output, accounted for uncertainty in those estimates, and checked the results using more samples and a different sampling method. These checks supported the overall findings.

The results suggest that cumulant propagation remains accurate under the small, fixed weight changes we tested. They do not yet show whether this holds for stronger changes, different directions of change, deeper networks, or changes learned through training.

Separate weak updates from benign geometry. In the first two weeks, retain the existing estimator and paired-seed measurement pipeline. Compare dense, coordinate-concentrated, and cross-layer-aligned updates with identical singular values. Begin with rank 2, depth 3, and norms 0.25, 0.5, and 1; vary several independently chosen orientations. Measure scaled error, output variance, and the change in the network’s input-averaged output. This checks whether apparent robustness merely reflects updates that barely affect the function.

Use the differences to choose the theorem. If orientation changes the width trend, identify the first affected layer and omitted cumulant. Start the proof attempt with rank-one, shallow polynomial networks, expanding Gaussian and deterministic contributions separately. Exact Gaussian moments in small polynomial examples provide checks. A counterexample would replace the unrestricted claim with a condition on singular-vector concentration or alignment. If no geometry effect emerges, vary rank and depth separately before widening the claim.

Test dependence introduced by training. Freeze the Gaussian base and train low-rank adapters on a synthetic teacher task. Compare learned adapters with orientation-randomized adapters preserving their singular values, and with independently constructed adapters whose spectra were fixed before initialization. Use repeated initialization and data seeds; record task loss and adapter norms throughout. Randomizing learned orientations retains dependence through the learned singular values, so it tests alignment rather than complete independence from initialization.

Change the algorithm only after locating a failure. A reproducible error increase tied to particular directions would motivate explicitly tracking their projected cumulants. Test that modification against the unchanged estimator at matched error and computational cost. Preselect evaluation widths and seed budgets, retain high-error networks, and repeat reference audits for any apparent failure.

I expect small weight changes spread across many neurons to preserve the observed decrease in error with width. Changes concentrated on a few neurons may behave differently. Success would mean proving when this accuracy is preserved, or finding a clear counterexample that shows why the claim needs narrower conditions.

Worse results at larger widths would motivate further investigation, but would not alone disprove the mathematical claim.

Our small training experiment found slower error reduction in both trained and untrained comparisons, so we cannot attribute it to training. If larger experiments also find no meaningful difference, I would focus on understanding fixed weight changes rather than developing a training-specific algorithm.

Risks and open questions

The perturbations may simply be too weak. Fixed rank and norm limit their total energy as width grows. Stronger norm settings and measured changes in the network function will distinguish this explanation from robustness to substantial structure. Nevertheless, these are the right things to focus on at the beginning.

Training comparisons may remain confounded. Matching rank and norm does not match the full spectrum, orientation, or task loss. Report these quantities and interpret each control narrowly; a gap alone does not identify its cause.

Three months should produce one carefully resolved result: a restricted polynomial error bound or explicit obstruction, supported by reproducible experiments and a controlled adapter-training comparison. Month one tests geometry and strength; month two develops the polynomial argument; month three evaluates learned adapters and writes the results. The existing parallel runner and reference audits reduce setup time. Required resources are a multicore workstation, occasional GPU access, and mentorship in cumulant expansions. If the proof or controls take longer, drop broad training sweeps and extra depths first. Transformer experiments, uniform-in-depth guarantees, and general claims of practical speedup remain outside this project.

Can the training dynamics of intractable nonlinear neural networks be approximated by those of n tractable linear neural networks? Can this approach yield tractable predictions of the evolving neural tangent kernel (NTK) and other relevant observables?

Much of what we know about neural network training comes from limiting models or specially structured models whose parameter dynamics are solvable. Real more complex networks need not satisfy their assumptions, and numerical trajectories rarely explain why a plateau ends or a representation changes. We propose a middle ground: replace one intractable flow with several explicitly solvable models, then control how their local errors accumulate. If successful, this could yield a finite-time map of learning regimes: which modes grow, which directions rotate, and which layers reduce loss, without assuming a fixed NTK throughout training. If unsuccessful, it would show how quickly nonlinear feedback destroys local solvability and identify architectures or phases where simulation or different reduced descriptions remain necessary.

Deep linear networks admit exact solutions in restricted regimes, elucidating plateaus and ordered mode learning [1]. However, linear input–output maps do not imply linear or integrable parameter flows, and constant forcing generally invalidates their conservation-based reductions. The NTK gives a fixed-kernel description in suitable limits [2], while lazy-training theory characterizes when nonlinear paths stay close to linearized ones [3]. Other parameterizations allow substantial feature learning [4], so a single kernel at initialization can miss changing representations.

There are also methodological precedents. Iterative linearisation repeatedly refreshes tangent models and empirically investigates feature updates [5]; exponential Rosenbrock integrators repeatedly solve locally linearized ODEs and analyze their accuracy [6]. Restricted linear-attention models yield some tractable dynamics [7], but do not provide a general local approximation theory for nonlinear architectures. What remains open is a criterion for useful analytic tractability: can we choose a surrogate family whose corrected dynamics are explicitly solvable, whose validity lasts long enough to reveal mechanisms, and whose error bounds transfer to observables of the original nonlinear network? Existing results do not establish this combination across most architectures.

Consider a nonlinear neural network whose learning dynamics we first approximate by a gradient flow on a time interval [0,T], with loss L and evolving parameters z(t). Write the original flow as $\dot z(t)=F(z(t))=-\nabla L(z(t))$. Suppose these dynamics are intractable, meaning that simple ODE methods cannot summarize them. Partition [0,T] into n intervals [tₖ₋₁,tₖ]. On each interval, choose a linear network from an explicitly solvable family with surrogate loss $\tilde L_k$. At a reference point aₖ, add a constant correction to match the original flow’s velocity:

This correction matches the velocity at aₖ and corresponds to adding a linear term in the parameters to the surrogate loss. The corrected system must itself remain explicitly solvable. Solve it on the interval, then restart the next surrogate at the computed endpoint. Denote the local approximation error by εₖ and account for its propagation through subsequent intervals. We then seek a finite-time error bound for the stitched trajectory $\hat z(t)$:

Under appropriate assumptions, the error should tend to zero as the partition is refined. For a prescribed error tolerance ε, our goal is to estimate the number of intervals n required. For relevant examples, such as a simple transformer, we seek a construction in which both n and ε are relatively small.

We can track the original network’s NTK along the parameter path. Under smoothness assumptions, parameter error can bound NTK error. We can also analyze effective rank, eigenspace rotation, residual-aligned learning rates and changes in activation or attention patterns.

If the training Jacobian and key backpropagation interactions vary slowly, we expect a few solved intervals [tₖ₋₁, tₖ] to reproduce loss and NTK evolution even when the kernel changes substantially overall. Local mode spectra may identify periods of rapid residual decay, alignment, or stagnation.

We would reject this direction if tractability consistently required intervals as short as standard integration steps.

Three months would produce one theorem with explicit assumptions, a reproducible implementation, and a benchmark that includes failure cases. In month one, we would derive solvable families. In month two, we would prove error propagation bounds and develop an adaptive interval rule, with a separate bound for any splitting approximation. In month three, we would compare the NTK and other observables and test one restricted transformer attention example.

Self-supervised objectives (SimCLR [1], BYOL [2], SimSiam [3], Barlow Twins [4]) reach similar benchmark performance. Do they get there through the same gradient dynamics? In shared linear models, what does each objective learn, in what order, how fast, and why?

Suppose all objectives learn the same features in the same order, and differ only in how fast each feature is learned. Then choosing an SSL loss is mostly a choice of speed and stability. Results transfer across objectives, and hyperparameters can be mapped between them.

Suppose instead they differ. Then we get a dictionary linking each mechanism (negatives, decorrelation, or a predictor with stop-gradient) to concrete dynamical effects. It tells us which features are learned first, late or never, and how initialization scale, step size, batch size and noise change that.

SSL encoders are the perceptual front end of many agents and world models. Predicting what they learn from how they are trained is a step toward understanding and controlling these systems.

Learning mechanics gives exact, ODE-level accounts of supervised linear and diagonal networks: sigmoidal, mode-by-mode learning [5], conservation laws that come from symmetries [6], the effect of initialization scale and imbalance, rich versus lazy [7, 8, 9], saddle-to-saddle dynamics in the vanishing-initialization limit [10, 11], the effect of finite steps and SGD noise on what is learned [12, 13, 14].

For SSL, dynamical results come one objective at a time: Tian et al. [15] analyse BYOL and SimSiam and how the predictor aligns with the input correlations. Simon et al. [16] solve linearized Barlow Twins from small initialization and find stepwise learning of eigenmodes. Jing et al. [17] and Ziyin et al. [18] study dimensional collapse through stationary analysis rather than trajectories. Littwin et al. [19] show that deep linear self-distillation, as used in JEPA-style training, favours high-influence features. HaoChen et al. [20] characterise contrastive optimum spectrally, but not how training reaches it.

No work applies the full toolkit to several SSL objectives in one common model. That toolkit covers: exact ODE solutions and conservation laws, vanishing-initialization limits, finite-step maps, SDE and Fokker–Planck descriptions of noise.

Without such a comparison we cannot tell which dynamical differences are intrinsic to the objectives.

• 1.Common model. Gaussian data with spectrum $\Sigma$; augmentations as noise with covariance $\Sigma_\mathrm{aug}$; a two-layer linear encoder $W = W_2W_1$ and its diagonal variant $u\odot v$; a predictor head for BYOL and SimSiam [2, 3]. Write InfoNCE (in the large-batch limit), Barlow Twins and BYOL/SimSiam as functions of the end-to-end map, and derive their gradient-flow ODEs.
• 2.Find the symmetries and conservation laws [6]. Ask which ones the predictor, stop-gradient and EMA break; EMA makes the flow non-gradient.
• 3.When the covariances commute, reduce to mode-wise ODEs. Solve them exactly (logistic or Riccati type [5, 16]) or derive time-scale asymptotics, and identify the quantity that orders the modes.
• 4.Vanishing initialization: rescale time by $\log(1/\alpha)$ and derive the limiting staircase of jump times and plateaus [10, 11].
• 5.Rich versus lazy: study initialization imbalance and kernel evolution [7, 8, 9], and give a mirror-flow description of the implicit bias where one exists.
• 6.Finite steps: write discrete maps for the conserved quantities, as in [12].
• 7.Noise: model SGD as an SDE with multiplicative noise and write the Fokker–Planck equation for the mode amplitudes. Derive when a weak mode is killed (stochastic collapse [14]) or effectively re-initialized [13]. In high dimension, pass to a continuity PDE for the density of mode amplitudes.
• 8.Test the predicted orderings, time scales and noise thresholds on small MLP and CNN encoders (synthetic data, CIFAR-10), tracking per-mode alignment and effective rank.

Prediction: from small initialization, every objective learns augmentation-invariant modes stepwise, in an order set by a per-mode augmentation signal-to-noise ratio. The objectives differ in three ways:

• 1.their mobilities (sigmoidal versus exponential onset);
• 2.their conserved quantities;
• 3.the lengths of their plateaus.

We also predict that minibatch noise acts multiplicatively on weak modes. Dimensional collapse can then arise as a stochastic-collapse effect, with a threshold set by batch size and step size. It should be stronger for objectives without an explicit decorrelation term.

We would drop or reframe the question in either of two cases: Already in the linear model, all objectives reduce to the same ODEs up to a rescaling of time. Real networks show neither stepwise ordering nor a noise-dependent loss of weak modes.

Month 1: Set up the common model and derive the gradient-flow ODEs for common SSL models. Find their symmetries, conservation laws, reduce to mode-wise ODEs, and obtain exact or asymptotic solutions in the commuting case, checked by numerical integration.

Month 2: Derive the vanishing-initialization staircase and the rich-vs-lazy behavior for all 3 objectives, and the finite-step maps. Build the SDE model of minibatch noise and carry out the Fokker–Planck analysis of weak-mode collapse for one objective. Start experiments on synthetic data.

Month 3: Test the predicted orderings, time scales and noise thresholds on small encoders (synthetic data, CIFAR-10). Assemble the comparison table and write up.

Resources: An agent that would help us handle the theoretical derivations and computations. The experiments need a single GPU, roughly 100–300 GPU-hours.

Does a transformer trained on a star free/regular language, by next token prediction, does it have an algebraic structure beyond what prediction strictly requires, and if so, when does that structure emerge (before or after generalisation)?

The general idea; claims that models "learn maths" or "world models" dont really separate learning the underlying structure from learning just enough to do the task. Formal languages make the difference clear.

If monoid structure appears

Networks build algebraic structure beyond what is strictly necessary for token prediction

That gives interpretability a known target to search for, instead of reverse engineering structure after the fact.

If only states appear

Prediction models learn the minimal sufficient statistic and nothing else

That's a general check before any "model learned this thing" claim

If the algebraic class predicts the mechanism

Formal language theory might predict internals, not just behaviour.

In safety terms, it's a toy demonstration of predicting generalisation from internals before deployment

State tracking [1]: transformers trained on permutations represent maps between states rather than states. But permutations form a group, so "learned the algebra" and "learned the minimal statistic" can't be distinguished.

Algebraic interpretability [2, 3]: networks learn group composition via irreps. The extension to monoids works only where structure is locally invertible. Aperiodic monoids have trivial maximal subgroups, so the group-character methods used in [2, 3] have nothing to find.

Computational mechanics [4]: transformers linearly represent belief states. This is geometric, not algebraic.

Formal language theory [5, 6, 7]: establishes which languages transformers can express, and which they length generalise on (C-RASP). These results are constructive/behavioural, afaik none look inside trained networks.

Grokking [8]: structure before generalisation, but shown on single tasks whose structured solution was identified after, never on a task family with known competing representations.

• 1.Choose languages

For small star free languages, compute the monoid and measure size vs automaton's states, over the segments the model will actually see.

Keep languages where the gap is large (because this is the place where ‘learns the algebra’ vs ‘learns just enough to predict’ make different enough predictions to actually find.

Pick some matched pairs: a language transformers generalise on, for example a language expressible in C-RASP (a counting based formalism whose languages transformers provably length generalise on) and one that isn't expressible in this way.

• 2.Training

Small transformers, next token prediction on a fixed training set, with held out contexts and longer test sequences.

Sweep training set size and weight decay to find a regime where generalisation is delayed

• 3.Test what the final model represents

Monoid vs state, are chunks of input grouped by their monoid element, or only by their effect on the current state?

Counterfactual probes, ie, can you read out what a chunk would do from states the model isn't in? (Only a monoid representation should support this.)

Algebraic checks, do segments the monoid says are equal (like repeated letters that stop mattering) get identical representations?

The control is random groupings of chunks with the same class sizes as the monoid partition, nested within the same state classes, so that any extra explanations can't come from just having more labels

Test causality by intervening on the structure and check that predictions change as the algebra says they should.

Predictions

Intermediate layers encode chunks by monoid element, not just their effect on the current state.

Counterfactual probes read out what a segment does from states the model isn't in, above random grouping controls.

Intervening on this structure changes predictions as the algebra dictates.

Over training, monoid structure appears before the loss drops from the excluded data.

Final layers compress it down to states.

The generalising language is represented via counters, the other via shortcuts that break on longer inputs. (See Yang et Al, [7])

Would change my mind based on

Would drop the monoid question if, across several languages, monoid class never explains chunk representations better than current state does, and counterfactual probes stay at control level. That result means prediction trained transformers learn only the minimal statistic, not algebra.

Drop the counter hypothesis if the generalising language shows no linear features, or both languages are represented the same.

A pipeline: language selection, training, ground truth labels, etc

An answer to whether monoid structure is represented and causally used, on 2–3 languages.

Hopefully, emergence timing relative to generalisation.

One matched C-RASP / non-C-RASP contrast.

Needs not that much compute, on the order of a few hundred GPU-hours (rough Claude estimate).

Beyond three months but potentially doable depending on capabilities acceleration with new models in the loop: group components and a Hankel rank grokking sweep.

For quanta with Zipfian use frequencies $p_k \propto k^{-(1+\alpha)}$ placed on a random prerequisite DAG, where a model of capacity $n$ learns the downward-closed set of $n$ quanta with the largest total frequency, is the parameter-scaling exponent exactly $\alpha_N = \frac{\alpha}{(1+\delta)}$, where $\delta$ is defined by the prerequisite closure of the $m$ most frequent quanta having size ~ $m^{(1+\delta)}$? In particular, does this recover $\alpha_N = \alpha$ whenever prerequisites rarer than their dependents (“frequency inversions”) are asymptotically sparse?

The quantization model of Michaud et al. assumes: “QH2 (ordered acquisition). Networks learn quanta roughly in order of decreasing usefulness, so a given model has learned the first m quanta of a fixed sequence.” (from “Physics of Deep Learning - Part III Exercises: Neural scaling laws,” Iliad Intensive, October 2026; c.f., [1])

If our claim holds, the scaling exponent is set by two measurable properties of the data, not one: the frequency tail $\alpha$ and the prerequisite-closure growth $\delta$. That has three consequences:

It explains why common skills can emerge late: their ancestors are rare.

It turns emergence forecasting into a closure-cost calculation.

It suggests a concrete data intervention: upweighting rare prerequisites lowers δ and should steepen scaling.

It could also account for the gap between the gradient-cluster frequency exponent (about 0.24) and the measured $\alpha_N$ (about $0.08$), which $\delta \approx 2$ would produce.

If it fails, optimal learners skip expensive frequent skills, or $\delta$ is ill-defined. Then independence is not a harmless simplification, exponents cannot be read off frequency statistics, and power-law scaling under dependencies needs a different combinatorial invariant.

No existing result computes a scaling exponent for quanta with prerequisite structure.

The quanta model [1] derives $\alpha_N = \alpha$ by assuming independence: capacity goes to the $n$ most frequent quanta. This is valid only if the top-$n$ set is closed under prerequisites.

Compositional multitask sparse parity [2] shows empirically that prerequisites re-order learning, so frequency alone does not set acquisition order. It measures no exponent and proposes no invariant.

The Resource model [3] simulates learning on arbitrary dependency graphs and reproduces serial “Domino” acquisition. It solves only the forward problem on small hand-built graphs and gives no asymptotics for random DAGs with power-law frequencies.

Leap complexity [4] makes hierarchy govern SGD time - but for the Fourier support of a single target, not a frequency-weighted population of skills.

Hierarchical-grammar learning curves [5] combine power-law rule frequencies with hierarchy. For next-token prediction, they find the exponent decouples from the frequency law, which is evidence that structure matters. Their hierarchy is a generative tree rather than a prerequisite closure, and no closure-growth quantity appears.

Skill-graph frameworks [6, 7] use skill dependencies or co-occurrence to explain emergence and order curricula, but derive no scaling exponents.

The missing object is $\delta$: how prerequisite-closure size grows with frequency rank, together with whether greedy closure is asymptotically optimal.

The first step is to construct three random-DAG ensembles over quanta ranked by frequency, $p_k \propto k^{-(1+\alpha)}$:

(E1) Monotone. Each quantum draws d parents from more frequent quanta. This is the null case, $\delta = 0$.

(E2) Inversion-tunable. Parent ranks are drawn $\propto j^{-\theta}$ over all ranks, so $\theta$ controls how often a prerequisite is rarer than its dependent.

(E3) Layered. Frequency decays geometrically with depth.

For graphs of up to $10^7$ nodes I would compute the closure size $C(m)$ of the top $m$ quanta by reverse BFS, then fit $\delta$.

Second, I would compute the true optimal ideal, not just the greedy closure. The Lagrangian problem, maximizing $\sum_{k \in I} (p_k - \lambda)$ over ideals $I$, is a maximum-weight closure problem solvable by min-cut (Picard). Parametric min-cut (Gallo–Grigoriadis–Tarjan) gives the full nested family as $\lambda$ varies, which traces the concave majorant of captured mass against $n$. Comparing this to the greedy curve tests directly whether greedy closure is asymptotically optimal, and that result fixes the statement of the theorem.

Third, I would prove the result for E2. Exploring the ancestry of a quantum backwards is a branching process in rank. Its offspring tail determines $\delta$, and a criticality threshold beyond which closures span the graph. The goal is matching upper and lower bounds giving $\alpha_N = \frac{\alpha}{1+\delta}$.

Fourth, I would extend the result to data. A quantum is learnable only if it and all its ancestors appear $\tau$ times, which predicts $\alpha_D$.

Fifth, I would build compositional multitask sparse parity with E2-generated DAGs at fixed $\alpha$, vary $\theta$, train MLPs, and compare the measured $\alpha_N$ to $\frac{\alpha}{1+\delta}$.

Predictions:

E1 gives $\delta = 0$ and $\alpha_N = \alpha$.

In E2, $\delta$ grows continuously from $0$ as inversions become more frequent, up to a supercritical threshold. Beyond it, ancestry explodes, $\alpha_N \to 0$, and the loss plateaus.

Parametric min-cut matches the greedy exponent: skipping a frequent quantum forfeits mass that cannot be recovered polynomially.

In networks, the measured $\alpha_N$ tracks $\frac{\alpha}{1+\delta}$ within $\pm 0.05$ for $\delta \le 1$.

What would make me drop the question:

The optimal ideal beats greedy closure by a polynomial factor in natural ensembles. Then $\delta$ is the wrong invariant, and I would pivot to characterizing the min-cut hull directly.

No natural ensemble produces power-law closure growth; closures are either linear or explode. Then dependencies add no new exponent.

Networks learn dependent subtasks without their prerequisites (shortcut solutions). Then the DAG is not enforced, and the empirical arm fails.

Link to full proposal: https://docs.google.com/document/d/1AP0cUpMR8pxEtTvv4JMUlps2z9Zgu9Bhe0yFmsO3p88/edit?usp=sharing.

Month 1: ensemble generators, closure computation, and parametric min-cut. This is CPU-only and needs networkx and a max-flow library.

Month 2: branching-process proofs for E2, the criticality threshold, and the data exponent.

Month 3: compositional sparse-parity experiments, about 200 small-MLP runs (roughly 50 GPU-hours), and the write-up.

Deliverables: a theorem for E2, numerical phase diagrams for E1–E3, and a controlled empirical test.

Needs: a mentor fluent in random graphs and branching processes, plus modest GPU access.

If it overruns, drop first the transformer runs, then the data exponent. The theorem and the numerics are the core.

At the edge of stability, gradient descent holds the Hessian's top eigenvalue at 2/η and progresses along less sharp directions. Can each less sharp direction's curvature and remaining distance to its minimum predict long-run training speed?

Training dominates ML's cost, yet no mechanistic account explains how fast loss falls or why loss curves look as they do. Central flows [1] made the edge-of-stability regime predictable but leave "the global rate at which training loss decreases" open. A spectral answer would make it predictable: if Hessian eigenvalues break up into log-spaced clusters, each loss drop is one cluster being fit, and the spacing between clusters sets when drops come. It would also show how step size changes that schedule. Overtrained models are harder to fine-tune [2], and alignment relies on fine-tuning. If the current curvature says the loss won't drop again for a long time, stopping now gives up little improvement and keeps the model fine-tuneable.

Full-batch gradient descent typically runs at the edge of stability: the top Hessian eigenvalue rises to 2/η and hovers there while the loss keeps falling [3]. Central flows predict the time-averaged trajectory accurately [1], and Berthier [4] derives them by separating fast oscillation, self-stabilization, and a slow drift that lowers the loss. None of this work, including follow-ups [5, 6], asks how fast that slow drift goes.

Two findings say where to look. Progress happens along the less sharp directions: at the edge of stability, updates restricted to the top eigendirections stop lowering the loss, while updates restricted to the rest train normally [7]. And those directions are sloppy [8]: trained networks' Hessian eigenvalues spread over many decades [9], in clusters [10]. For linear models, eigenvalues plus each direction's distance to the minimum determine the loss curve [11]. We know of no such account at the edge of stability, where the spectrum changes during training.

Separately, overtrained models fine-tune worse [2], and models lose the ability to keep learning as they lose curvature directions [12], but these effects are measured after the fact, not forecast from the spectrum.

We measure each less sharp direction's curvature and remaining distance (how far the weights are from the valley's bottom along it), predict the loss curve, and test the prediction:

Toy models. Losses with log-spaced eigenvalue clusters below the top, chosen distances, and a cubic term so gradient descent settles at the edge of stability [13]. The prediction, $\mathrm{loss}(t) \approx \min + \sum \tfrac{1}{2} h d^2 \exp(-2\eta h t)$ for curvature h and distance d, should match gradient descent and the central flow.

Curve collapse. With the central-flows code [1], train its small networks (e.g., a vision transformer on a CIFAR-10 subset) at five step sizes. Each less sharp direction moves at a speed proportional to η × its curvature. So if those directions set the rate, loss plotted against η × steps should collapse onto one curve at long times. Whether it does is the first result.

Snapshot prediction. At checkpoints, estimate eigenvalues below the top with Lanczos (using only Hessian–vector products), started from the gradient. Since gradient = curvature × distance, this gives each direction's distance. Predict the remaining loss curve and compare with gradient descent and the central flow.

Diagnose failures. Do the less sharp eigenvalues rise toward 2/η, or does the oscillations' push toward flatter regions [1] act on the less sharp directions?

Stopping. Stop some runs where the snapshot predicts a long plateau and continue others. Fine-tune both on held-out classes and test whether the early-stopped models fine-tune better at little cost in loss.

Toy models: loss falls in stages, about one cluster per decade of training time, each ending after roughly 1/(η × cluster curvature) steps. The slow drift of [4] splits into one timescale per cluster.

Networks: the snapshot should hold for a while, then drift. We guess the less sharp eigenvalues creep toward 2/η, as top eigenvalues do [3], speeding those directions up. If so, curves at different step sizes won't fully collapse, and the mismatch measures how much edge-of-stability dynamics reshape the spectrum.

Stopping: runs stopped at a predicted long plateau should fine-tune better than runs trained past it, at small cost in loss.

What would change our minds:

If curves collapse and the snapshot holds throughout, the rate is a simple spectral quantity and the central flow matters only locally.

If early stopping doesn't help fine-tuning, fine-tuneability isn't tied to the loss curve's stages.

Month 1: toy models with designed spectra, confirming the snapshot prediction where the answer is known, plus curve-collapse tests across five step sizes on the central-flows paper's small networks. Month 2: snapshot predictions at checkpoints on two or three of those networks and, where the predictions drift, identifying which of the two causes in step 4 is responsible. Month 3: the stopping test (early-stopped vs. longer-trained models, fine-tuned on held-out classes) and a workshop paper, with code for measuring curvatures and remaining distances along training runs.