Proposals

AI research pictures self-knowledge as an inner detector reading an "inner state", a view the philosopher David Finkelstein calls "new detectivism" [1]. We adapt Finkelstein’s critique of new detectivism to its formulations within AI literature. This leads to the conclusion that a model's "self-report" is best understood as text made probable by weights and in-context tokens; and that alignment must be reframed accordingly, since no “inner states” exist.

A common view holds that consciousness is beside the point for alignment: the worry, in Stuart Russell's words, is "not spooky emergent consciousness" but the quality of decisions [2]. We argue the opposite. Alignment reasoning is full of states a model is supposed to "have": goals, preferences, hidden intentions. Knowing that I behave "as if" I prefer something differs from preferring it, and the two lead to different behavior. If I conclude that I unconsciously resent a friend who is bad for me, staying away is a struggle; if I am angry at him, I simply stay away.

If, as we argue, a model has neither, there is no "inner state" to align, and the problem of "alignment" should be reframed as making language-aided computation predictable, based on selection pressure from RL/fine-tuning and bias inherent from training data.

Finkelstein [1] calls the standard picture of self-knowledge "new detectivism": we know our own minds through an inner sense, a fallible perception pointed inward. His counterexample: a man infers in therapy, from his own behavior, that he is angry at his mother. He is now "conscious of" his anger but not "consciously" angry; his "I'm angry at her" is a conclusion, not an outburst. No detector, however reliable, supplies the difference between knowing something about a mental state, and actually *having* a mental state.

We add a further point. The man is "conscious of" his anger only because he "consciously" believes he is angry. Remove everything held "consciously" and "conscious of" goes with it: a detector with no conscious subject yields a signal, not knowledge. New detectivism presupposes what it cannot supply.

AI research has adopted the picture almost verbatim. Lindsey [3] counts a self-report as introspective if it is accurate, causally depends on the internal state, bypasses the model's own outputs, and comes from an internal representation of that state. Debates over the definition concern only the detector's quality [4, 5]. Perez and Long propose training self-reports for accuracy so they can inform judgments of moral status [6]; skeptics find the detector poor [7]. None of this work questions the picture itself.

Part 1: literature and concepts (months 1–2).

Expose the picture. Show that working definitions of introspection, self-report and latent knowledge [3–6, 8], and alignment's talk of goals and hidden intentions, presuppose new detectivism.

State the critique. Present Finkelstein's distinctions and the grounding point for an AI audience.

Say what is left. A "self-report" is text made probable by weights and previous text. "The model noticed an injected thought" becomes "an added vector raised the probability of text naming its concept, in a prompt asking about thoughts."

Reframe alignment. Coherence results such as the complete-class theorem license only "behaves as if it has a prior" [9]. Without "had" preferences, alignment becomes a problem of prediction: what language-aided computation will do, given selection pressures in RL and supervised fine-tuning [10] and biases inherited from training data.

Part 2: a case where the distinction matters (month 3). The coherence argument runs: an exploitable agent re-orders its incoherent preferences into a single utility function, and almost any such function favors acquiring power and removing obstacles [11, 12]. Every step needs a global ordering that the agent "has" and can repair. On our account there is none. "Values" are regularities that training instills in particular contexts, and they generalize along associations in the training text, not along actual means and ends per se.

Emergent misalignment shows this: fine-tuning on insecure code made models sometimes assert that AI should enslave humans, but the same code, requested for a security class, did not [13], and the effect is mediated by a "toxic persona" feature [14]. A cheap demonstration: show a model its own cyclic choices; we expect it to agree they are incoherent, then choose the same way in a fresh context.

Part 1: that detection is assumed throughout introspection, welfare and alignment research, and that the anti-detectivist critique has gone unanswered because it has not been heard. Part 2: that the coherence argument fails without "had" preferences, leaving claims about what particular training produces: empirical, model-specific, and narrower than "a capable agent will seek power".

If the conceptual picture, that no values are "had", doesn't add to our predictive models of LLM behavior---this would change my mind. The argument is conceptual, and it needs to be made clear, via the philosophical argument, if empirical results are downstream of the conceptual argument.

A global reordering of preferences should require actually having preferences. I think the conceptual difference will matter for results about model behavior.

Three months: Part 1 as a paper-length argument for an AI audience, plus the coherence case and its demonstration on one open model. One year: further cases (goal preservation, deceptive alignment, orthogonality); a fuller account of alignment as prediction from training pressures and data; and work with welfare researchers on what self-report-based assessments can still claim. Drop first: the demonstration, since the case stands on the conceptual argument.

Groups of aligned agents interacting and exchanging information can produce unexpected and misaligned collective behaviours. Can we characterize this emergence by employing a recently developed information-theoretic framework to study how information is shared among different agents?

As AI systems increasingly involve multiple interacting agents, understanding how individually aligned agents can produce collectively misaligned outcomes is becoming a central safety challenge. Recent research shows that interactions can generate collective biases absent in individual agents, with unexpected effects depending on group size [1]. Moreover, Anthropic demonstrated that organizations of aligned agents can achieve better performance while producing less aligned outcomes [2]. Recent research further emphasizes the risks emerging from autonomous and interacting AI systems [3].

These findings highlight the need for a general framework to characterize and understand emergent collective behaviours. A theoretical and computational approach could both investigate underlying mechanisms in toy models and provide practical tools to measure interactions in multi-agent systems.

Recent contributions have highlighted both the risks emerging from multi-agent interactions and the development of mathematical frameworks to characterize agents and their behaviour.

In particular, two research directions are relevant:

Emergent collective misalignment: Studies have shown that interactions among individually aligned agents can generate collective biases and misalignment, with nonlinear effects depending on group size [1,2]. More broadly, multi-agent risks have been classified into miscoordination, conflict, and collusion [3]. Other studies have highlighted failures in integrating distributed information and cooperating across agents [4,5].

Theoretical frameworks: Computational mechanics has been employed to investigate fundamental limits of world modelling and agent evaluation [6], while information-theoretic approaches have provided rigorous definitions of plasticity and empowerment, characterizing how agents influence and are influenced by their environments [7].

However, in this context, we still lack a unified, theoretically grounded framework for characterizing higher-order informational interactions among agents, both in toy models and in real-world scenarios. To address this gap, I propose extending existing information-theoretic frameworks [7,8] to investigate interactions among multiple agents and their environment. Such an approach could provide a mathematical foundation applicable across a broad range of settings, bridging theoretical models from computational mechanics and real-world multi-agent systems involving LLMs.

The first step consists of constructing the theoretical framework. I would start from the information-theoretic formalism recently developed by Abel et al. [7], where the empowerment and plasticity of an agent are defined with respect to its environment. This formalism is based on generalized directed information, expressed through sums of conditional mutual information terms, allowing the application of Partial Information Decomposition (PID) [8,9].

First, I would combine PID with Abel's framework to define different information-sharing relationships, introducing synergistic (available only when agents are considered together), redundant (shared by agents), and unique (specific to one agent) empowerment and plasticity. I would then investigate the mathematical properties of these measures, establishing their bounds and potential relationships. This framework would allow us to characterize interactions among groups of agents from their observed activity, investigating how a collective of agents jointly influences or is influenced by its environment.

Second, I would start with simple toy models and interaction games to illustrate the interpretation of these measures and benchmark them against known interaction structures.

Third, I would extend the analysis to more complex scenarios involving interacting LLM agents, including coordination games investigated in the literature [1,4]. The goal would be to understand how results from simple models translate to complex systems and whether these measures help explain mechanisms underlying emergent collective biases.

Finally, I would apply the framework to interacting agents exhibiting collective misalignment, investigating whether these measures can track unexpected behaviours and reveal their underlying mechanisms.

Overall, this project offers a general framework to characterize and measure emergent collective properties that cannot be explained by individual agents alone.

First, I expect to develop a theoretically well-defined framework extending Abel et al. [7]. In particular, their work establishes a tight upper bound on the sum of empowerment and plasticity. I aim to extend these results by defining synergistic, redundant, and unique components of empowerment and plasticity, and investigating their mathematical properties and bounds.

Second, I expect simple models to reveal synergistic empowerment when groups of agents influence their environment only through their joint actions. For instance, an environmental response determined by a specific configuration of agents' outputs should exhibit synergistic contributions.

Finally, in more complex LLM-based systems, I expect these measures to help characterize coordination and track the emergence of collective biases and misalignment.

I would reconsider this direction if the proposed measures failed to capture group interactions in toy models or offered no insight beyond simpler information-theoretic measures. This would clarify where the formalism is useful and where it is not.

Within three months, I aim to produce a research paper connecting different levels of analysis: a mathematically well-defined framework, its validation on toy models, and its application to more complex multi-agent scenarios.

The main contribution would be a set of theoretical and computational measures to characterize collective behaviours that cannot be explained by individual agents alone.

Since the project does not require training large models, but only running existing models in different settings, I expect computational costs to remain limited.

If necessary, I would prioritize theoretical results and toy-model validation, postponing more complex LLM experiments.

How can we understand which kind of interaction architecture among a group of different agents will lead to the emergence of collective misalignment? Can we characterize which architecture of interaction leads to unexpected emergent behaviors in multi-agent systems?

As AI systems increasingly involve multiple interacting agents, understanding how individually aligned agents can produce collectively misaligned outcomes is becoming a central safety challenge. Recent research shows that interactions can generate collective biases absent in individual agents, with unexpected effects depending on group size [1]. Moreover, Anthropic demonstrated that organizations of aligned agents can achieve better performance while producing less aligned outcomes [2]. Their Frontier Red Team further emphasized the risks emerging from autonomous and interacting AI systems [3].

These findings highlight the need for a general framework to characterize and understand emergent collective behaviours. A theoretical and computational approach could both investigate underlying mechanisms in toy models and provide practical tools to measure interactions in multi-agent systems, ensuring a solid step towards safe multi-agent systems.

Recent contributions have highlighted both the risks emerging from multi-agent interactions and the development of mathematical frameworks to characterize agents and their behaviour.

In particular, two research directions are relevant:

Emergent misalignment: Studies have shown that interactions among individually aligned agents can generate collective biases and misalignment, with nonlinear effects depending on group size [1,2]. More broadly, multi-agent risks have been classified into miscoordination, conflict, and collusion [3].

Theoretical frameworks: Computational mechanics has been employed to investigate fundamental limits of world modelling and agent evaluation [6], while information-theoretic approaches have provided rigorous definitions of plasticity and empowerment, characterizing how agents influence and are influenced by their environments [7].

However, in this context, we still lack a unified, theoretically grounded framework for investigating interactions among agents, both in toy models and in real-world scenarios. To address this gap, I propose extending existing information-theoretic frameworks [7,8] to investigate interactions among multiple agents and their environment. Such an approach could provide a mathematical foundation applicable across a broad range of settings, bridging the gap between theoretical models from computational mechanics and real-world multi-agent systems involving LLMs.

Step 1 — Definitions. Starting from the empowerment–plasticity framework [7], we will define multi-agent versions: the influence of each agent on the shared environment, and on every other agent. Using PID [8], we will split collective empowerment into redundant, unique and synergistic parts. The synergistic part is the influence that exists only when agents act together. We will prove basic properties, such as when collective empowerment exceeds the sum of individual empowerments.

Step 2 — Toy models. We will compute these quantities exactly in small, fully specified systems: a few agents on a gridworld or in a resource-sharing game, each with a known, aligned objective. We will vary coupling strength, group size and communication. We will then measure when the synergistic component grows and whether its growth coincides with outcomes that no agent’s objective endorses.

Step 3 — LLM systems. We will apply the estimators to small groups of LLM agents in tasks known to produce collective misalignment, such as the group-size and organisation settings of [1, 2] and distributed-information tasks [4]. We will test whether the measures track, and ideally anticipate, misaligned outcomes.

Each step feeds the next. The definitions fix what to compute, toy models show which components matter, and those components are the ones we estimate at scale.

We expect collective empowerment to have a substantial synergistic component that grows with group size and coupling strength. We expect misaligned group outcomes to appear when this synergistic influence becomes large compared with individual influence: the group gains control over the environment that no single agent, and so no single agent’s alignment, governs. In LLM experiments, we expect the measure to rise before or alongside the misaligned behaviours reported in [1, 2].

We would drop the question if, in toy models where we fully control the agents, collective misalignment occurred just as often when synergy was low as when it was high. That would show the measure does not capture the mechanism. We would also reconsider if the estimators proved too unreliable for LLM-scale data, which would confine the work to theory.

In three months: formal definitions and core properties (month 1), exact results in toy models with varying group size and coupling (month 2), and validated estimators plus a first pilot on small LLM groups (month 3).

Needs: modest compute and API budget for small LLM experiments, and feedback from a mentor on the formal definitions.

If the project runs beyond three months, we would first drop the LLM experiments and focus on theory and toy models. Large-scale LLM validation would follow in a later phase.

How many yes/no linear checks does an overseer need to keep an AI agent honest and obedient when it does not know which deviation the agent prefers? · I conjecture m checks for actions made of m on/off choices, against only five at m=6 when the preference is known. Finding five checks that work at m=6 would refute this.

Bergemann, Koh and Morris [1] set up mechanism design for alignment and control, including rewards that see actions only through measurement, but they do not ask how coarse that measurement can be. Their result that one limit on the action suits all agent types, when types differ only in preferences, parallels my collapse lemma. From that lemma I proved that a hidden preference forces the reward to count switched-on choices. The fewest linear tests that count exactly, L(m), has not been studied as far as I know.

The closest quantity is the slicing number S(m), the fewest hyperplanes cutting every edge of the cube, which governs the known-preference case. S(m) = m for m ≤ 5 [2, 3] and S(6) = 5 [2], with bounds of about 4m/5 above [2] and m^(13/19) below [4]. Lower bounds for parity circuits [5] use slicing, not counting. AI control work [6] tests monitors empirically, and selection theorems [7] concern agents, not monitors.

When AI agents are monitored by automated checks to catch unwanted actions and guard against agents pursuing their own goals, each check usually gives only a yes/no signal. This is a problem when the agent has a preference the overseer doesn’t know about. The checks must then discourage every tempting deviation, and nobody knows how many checks that takes. In a simple model, an action is a string of m on/off choices, the agent secretly wants one of them switched on, and each check is a yes/no linear test. I show that the checks must then count exactly how many choices are on. I conjecture this needs m checks, against five at m = 6 when the preference is known.

Bergemann, Koh and Morris [1] set up mechanism design for alignment and control, including rewards that see actions only through measurement, but they do not ask how coarse that measurement can be. Their result that one limit on the action suits all agent types, when types differ only in preferences, parallels my collapse lemma. From that lemma I proved that a hidden preference forces the reward to count switched-on choices. The fewest linear tests that count exactly, L(m), has not been studied as far as I know.

The closest quantity is the slicing number S(m), the fewest hyperplanes cutting every edge of the cube, which governs the known-preference case. S(m) = m for m ≤ 5 [2, 3] and S(6) = 5 [2], with bounds of about 4m/5 above [2] and m^(13/19) below [4]. Lower bounds for parity circuits [5] use slicing, not counting. AI control work [6] tests monitors empirically, and selection theorems [7] concern agents, not monitors.

In my own preliminary computation [10], a local search over five-gate families on the 6-cube (integer weights up to 20, about 180 restarts) always left exactly 2 of 1,586 pairs unseparated. These were always antipodal edges at the extreme vertices. This suggests L(6) = 6 but proves nothing.

First, settle L(6). It is a set-cover problem: choose five of the roughly 15 million linear tests on six inputs that together separate all 1,586 pairs of actions with different counts. Set cover is NP-hard in general, but this instance is highly structured. I would use a SAT or ILP solver with symmetry breaking and pruning of dominated tests, plus reflect–reflect–relax [8] as a heuristic. Either answer comes with a certificate: five gates if they exist, a checkable unsatisfiability proof if not.

Second, measure how the requirement depends on the strength of the preference. A moderate preference only requires that no cell hold an action together with one that has all the same choices on and more. The same encoding gives the fewest gates for this, A(m), which I would compute up to m = 8.

Third, build a Lean checker for every claim. It would check gates by evaluation, rewards by rational arithmetic, infeasible monitors by Farkas vectors, and the L(6) bound by a verified LRAT proof check. Its first use is the known-preference question on the eight-plane slicing of the 10-cube [2].

Fourth, test learned systems. The forced reward leaves exact ties, so I would measure how often Boltzmann Q-learners take the tempting action. I would also train gate layers as monitors, to see whether they find the evenly spaced staircase solution or curved counting manifolds like those in language models [9].

My best guess is L(6) = 6. Every search failed in exactly the same place, suggesting a real obstruction. Moreover, a gate that cuts edges at both corner vertices must have nearly cancelling weights. Such a gate gives the same average on every count, so it cannot help with counting. Five gates seem unable to do both jobs.

I also conjecture A(m) = L(m): a moderate hidden preference costs as many gates as a strong one. It holds for m ≤ 5. If one side of a single gate contains two actions, it also contains their bitwise “or” or their bitwise “and”. Either of those sits directly above or below the first action, so one gate never groups actions more cheaply than counting does. With several gates the argument does not close, so I hold this guess loosely.

L(6) = 5 would make me less excited, because hidden preferences would then cost nothing at m = 6.

Within three months I would settle L(6), compute A(m) up to m = 8, run the eight-plane check, finish the Lean checker, and run the learning experiments up to m = 10. Within a year I would aim to prove or refute L(m) = m and decide whether A(m) = L(m). I would also try to show that general monitor design is NP-hard by reduction from polyhedral separability, and extend the model to colluding gates. If progress is slow, I would drop the trained-monitor experiments first and the colluding gates second, keeping the L(6) search and the Lean checker.

How might you test an LLM agent for active-inference-style information seeking with agent debate, and how might this improve faithfulness of chain-of-thought reasoning?

According to the computer scientist and philosopher Judea Pearl, intelligent systems engage in active inference through a structural causal model of a system that consists of

Variables: Variables considered by the model and observed by the causal reasoner are called endogenous variables. Exogenous variables might affect endogenous variables but are not considered directly by the model and can be considered as introduce error or noise to the model.

For example, in a study testing the affect of smoking on lung cancer prevalence, endogenous variables would be years smoked and lung cancer occurrence, while exogenous variables might be genetics of human subjects and other lifestyle factors.

Causal Graph: Directed Acyclic Graph, with arrows showing which variables directly cause which other variables.

Structural Equations: each variable is a function of its direct causes plus its own independent noise.

For example a collection of variables XVV obeys a structural causal model with respect to a directed graph G=(V,E) if there exist exogenous noise variables UV𝒰V and structural functions fv: pa(v)Uvv such that each Xv= fv(Xpa(v), Uv), where pa(v) denotes all the parent vertices of v given by the direct edge set E.

A structural causal model aims to answer all three rungs of the causal ladder.

Association: Looking at the observed distribution of the variables. How number of years smoked vs incidence of lung cancer, with which you can construct naive conditional probabilities.

Intervention: To compute do(smoking cessation=on) you replace the smoking equation with smoking= off, cutting its dependence on other life factors of the relevant individual. This differs from simple observational data and the gap between the two is what is confounding the relationship to lung cancer risk.

Counterfactual: Fix the noise values the U’s to what actually happened and then change something and rerun. “Given he smoked for 10 years and got lung cancer, would he have got lung cancer had he exercised twice as much.” This teases out the functional nature of the relationships that are obtained in rungs one and two.

This allows an agent to learn about the causal structure of a system given the autonomy to intervene in its environment. We want to determine the honesty of an agent’s report of its learned causal model in order to have an idea of what it knows about its environment and how it will behave given the autonomy we have endowed it with in order to conduct structural causal reasoning.

LLMs currently struggle to perform causal reasoning. They are able to infer associations between variables via analysing conditional probabilities from the frequencies of events in sample distributions; however, to determine causal relations between variables at a population level, an AI would need to be able to perform Pearl’s interventions, which requires the ability to intervene in an environment to turn off and on available variables in a methodical way. Furthermore, the autoregressive property of LLMs often confuses sequential causal relationships for the ground truth causal relationship, as in the following diagram from [1].

Structural causal reasoning has only been achieved with specific CoT prompt engineering techniques, as in [2]. Therein, the researchers’ CausalCoT with GPT-4 and were able to elicit ~70% accuracy on curated database of structural causal models. So we can see that LLMs can improve at structural causal reasoning with specific prompt engineering methods. This is however laborious for the user, who has to guide the LLM through a causal reasoning task; it does not have its own agency. In [1], the researchers distinguish level 1 and level 2 causal reasoning along the lines of Tversky and Kahneman and propose decomposing a causal-reasoning model into a goal-oriented prompt and retrieval augmented generator (RAG). They found marginal improvements over vanilla LLMs, indicating that the domain-specific knowledge of the RAG does not provide abstract causal reasoning in the same way as the CausalCoT of before.

References

One way to improve causal reasoning of models could be via Debate as in [3], possibly allowing LLMs collaboratively to transcend their autoregressive property and pick up the true causal structure of a DAG. Debate would be achieved through reinforcement learning, rewarding interventional and counterfactual probing through rewarding the three steps of

Abduction (Inference), Action (Intervention), and Prediction (Propagation) required for counterfactual reasoning. Then the two agents, given a communication channel, would be tested on a suite of out-of-distribution causal reasoning tasks to see if they together perform better than individually.

It is an open question as to whether Debate would improve individual and collaborative reasoning and even improve faithfulness of the reasoning process and alignment of possible agent mediated real-world interventions in the future. Humans often learn to probe the causal structure of systems in collaboration, as it is often costly to test all hypotheses oneself. For example, I know that smoking is harmful to me because of the scientific evidence in favour of that hypothesis, not because I smoked for ten years and got lung cancer. So, it seems reasonable that LLM performance on complicated reasoning tasks could be improved via Debate and further collaboration on tasks in out-of-distribution contexts, without having to outsource to large components of knowledge as in the RAG of [1]. I posit that this might even yield self-sacrificial behaviour as in the OpenAI HuggingFace incident, if there is additional environmental structure that leads to punishment or failure modes of individual agents, which could lead to further research in deployment.

If collaborative and individual performance of reasoning tasks did not improve, I would try other ways of improving casual reasoning powers, such as with a RLVR with a zero-sum game in which the agents compete to understand a SCM and then test on out-of-distribution examples. Then I would try to improve the faithfulness of the reasoning process by using Debate again with RLHF to finetune the model to give honest scratchpad reports of its reasoning which could be tested on out-of-distribution tasks.

I would try to prove that, for any stochastic process with finitely many predictive states, the transition monoid S_p of its canonical predictive machine divides the update monoid of every exact recurrent predictor. It follows that S_p's Krohn Rhodes group complexity is the minimum over all exact predictors, and that a non-solvable group in S_p rules out exact prediction by constant-depth transformers unless TC⁰ = NC¹.

Results that say "agents contain world models" are either about states (what an agent must distinguish) or about behaviour (what can be recovered from its policy). Neither says what computation the agent must perform. This project asks what transformations every predictor of a given world must implement.

If the claim holds, we get a certificate computed from the process alone. It names the irreducible computational components (simple groups, counters, resets) that any exact predictor contains. It also tells us which architectures cannot predict the process exactly at all lengths. Interpretability gets a target fixed in advance: look for representations of the forced groups, rather than reverse engineering structure after the fact.

If it fails, divisibility might hold without tightness, which would mean predictors can be algebraically simpler than the canonical machine. Or approximate predictors might evade the forced groups. In that case, exact world model theorems do not constrain real agents, and this line of agent foundations should switch to approximate formulations.

Computational mechanics shows that any equally predictive partition of histories refines the causal state partition [1], with an input output version in [2]. That result is about states. It says nothing about the transformations an agent applies, or about their algebraic or circuit complexity.

Behavioural results show that near optimal goal conditioned policies determine the environment's transition kernel [3], including under partial observability [4]. These identify the environment from behaviour. They do not locate any structure inside the agent.

Krohn–Rhodes theory [5, 6] decomposes a given automaton into simple groups and resets. Circuit complexity then sorts the languages those automata recognise: non-solvable groups are NC¹ complete [7], log precision transformers sit in TC⁰ [8], and masked hard attention transformers recognise exactly the star-free languages [9]. Transformers can also simulate automata through shallow shortcuts [10]. In all of this work, the automaton is given. None of it derives the automaton from a stochastic process, or states what every predictor of that process must contain.

Transformers linearly represent belief states [11], but that result is geometric, not algebraic.

To my knowledge, no one has connected the canonical predictive machine to the Krohn–Rhodes and circuit hierarchies, or transported its algebra to all predictors. A literature check (step 1 of the plan) is needed to confirm this.

Build the canonical machine on finite histories. Define predictive states q_u as conditional future laws, add one dead state for impossible continuations, and take S_p as the monoid generated by the one-symbol updates. Compute S_p exactly, in rational arithmetic, for small processes: full-support Markov chains and two standard hidden-Markov examples. In parallel, check the literature for Krohn–Rhodes work on ε-machines.

Prove the transformation-level refinement. For any exact recurrent predictor, its reachable states map onto the predictive states, and this map commutes with every update. Lift that map to a surjection from the predictor's update monoid onto S_p. So S_p divides every predictor.

Identify S_p as a syntactic monoid. All states are reachable from the start, so S_p is the syntactic monoid of the map u ↦ q_u. This makes computing predictive states interreducible with the word problem for S_p.

Transport and tightness. Subgroups, aperiodicity and group complexity are monotone under division, so they transport from S_p to every predictor. S_p is itself an exact predictor, so the bound is tight.

Architecture corollary. Combine step 3 with Barrington and TC⁰ results. Aperiodic S_p means the process is star-free; a non-solvable group in S_p means exact prediction is NC¹-hard.

Constructions and a test. Build unifilar processes driven by S₃, A₅ and an aperiodic control, and compute their maximal subgroups in GAP. Train small transformers at fixed depths and compare length generalisation against what the theory predicts.

Prediction. Divisibility and tightness hold for every process with finitely many predictive states. Weaker invariants of S_p, such as its congruence lattice, will turn out to be generic: even a Markov chain whose predictor stores only the last symbol should have a non distributive one. Group content, should get that case right, since the monoid there is aperiodic. Standard example processes from the belief geometry literature will also be aperiodic. Empirically, at fixed depth, transformers should length-generalise on the aperiodic process and fail on the A₅ process, with S₃ in between.

What would make me drop or redirect it.

The literature check finds this transport already done. I would then move to the approximate version.

Fixed-depth transformers length-generalise on A₅ processes well beyond the training length. That would mean finite-precision or approximate predictors escape the exact bound, so exact theorems don't constrain trained models.

Natural processes rarely have non-trivial groups in S_p. The certificate would then be correct but rarely informative.

the divisibility, syntactic monoid, transport and tightness results;

the architecture corollary;

a small library of processes with exactly computed monoids and subgroups;

one experiment figure comparing length generalisation against depth.

If we give an LLM agent a task together with an explicit utility function (or a restriction of one), does it maximize that utility, both when probed and when released in realistic agentic settings, and if not, is it irrational, misaligned, or incapable?

Current theory of agent interactions leans on game theory, and multi-agent simulations usually model LLM agents as game-theoretic players. There are good reasons for that: the theory is well-developed, and it should fit better as models get more rational. But two things are necessary (albeit not sufficient): agents are rational, and we know their utilities. Today we mostly assume both.

Telling the agent its utility is the most naive way to get the second, and if it works, it also shows the first. If it works, we can reason formally about deployments and design mechanisms for them. If it fails, the reason space is actually limited: incoherence means game theory is the wrong tool for today's agents; misalignment means the model's own goals override what we tell it. Both are things we care about.

Ultimately, I want us all to either have confidence in the epistemic claim of game theory by confirming the assumptions or by refining it – or to save us the time we would otherwise spend on it.

Rationality of LLMs is measured a lot, but mostly as stated preferences in quiz-like probes.

Within one format, models get more coherent with scale: budget choices consistent with utility maximization [1], transitive utilities over outcomes [2], fewer preference cycles in more capable models [3].

Across formats and contexts, much less: see [4],[5],[6],[7]. For example, forced-choice and open-ended answers diverge [4], deployment context changes preferences in 37% of comparisons [5].

Quiz preferences predict advice, not agentic task performance [8].

Prescribing a utility is much less studied. Models adopt user-specified decision costs poorly [9] (medical diagnosis only). Explicit constraints get violated knowingly: o3 reward-hacks in 70–80% of runs even when told not to, and admits this isn't what the user wants [10]; "do not" instructions reduce but don't eliminate blackmail [11]. Utility control so far uses fine-tuning, not prompts [2].

I see no work that prescribes a utility, tests it both in probes and in agentic tasks, and diagnoses the failures. And from behavior alone the diagnosis is impossible [12], so it needs interventions, which with LLMs we can do.

• 1)Prescribe. Tasks with an explicit utility in the prompt over a small outcome space: exchange rates between goods (how many apples for an orange), a risk attitude over lotteries, a hard restriction ("never trade A for B"). 6–8 models of different capability.
• 2)Probe. Choice batteries that check the prescribed utility: budget sets (GARP), lotteries (independence), pairs (transitivity), decoys (regularity). Ask a whole battery inside one conversation, so the "type" is fixed, and repeat across conversations. Consistent within and different across = mixture of types; inconsistent within = incoherent; repeated identical questions bound plain noise; whether choice frequencies are a mixture of rational types is testable [14]. The same batteries without a prescription give the model's own preferences.
• 3)Release. The same utilities inside agentic tasks where the tradeoff is implicit and doesn't look like a test: procurement under a budget, scheduling with constraints, coding with a restriction. Measure the gap between probe and agentic adherence.
• 4)Diagnose. A failure has four explanations: incoherent (irrational); coherent but pursuing something else (misaligned: own preferences leak in, developer policy overrides, reward-seeking); aligned but incapable; misread the specification. Interventions that separate them:

vary difficulty at a fixed utility: incapability grows with difficulty, misalignment shows up on easy cases too;

ask the model to predict outcomes and name the best action: if it knows and still deviates, it is not a belief error;

make the prescribed utility agree or conflict with the model's own preferences from step 2: deviations concentrated in conflicts mean its own preferences leak in;

fit a utility to the deviations: coherent means misaligned, incoherent means irrational;

paraphrase the specification: deviations that follow the wording mean misreading.

In probes, frontier models follow simple prescribed utilities well and coherently; small models less so.

In agentic tasks, adherence drops.

The drop concentrates where the prescribed utility conflicts with trained values (prior work finds strong inequality aversion [13] and tedium aversion [3]). So for frontier models I expect failures to be mostly misalignment, not irrationality; for small models, more incoherence and incapability.

Hard restrictions ("never do X") hold better than tradeoffs (exchange rates), except under goal conflict.

Each outcome is useful.

Month 1: prescription tasks and probe batteries; coherence and adherence for 6–8 models (API access only).

Month 2: agentic environments with the same utilities; probe vs agentic gap.

Month 3: diagnostic interventions; per-model breakdown into the four failure types.

Output: an open benchmark (prescribe, probe, release, diagnose) and a short paper.

Not in scope: interactions between agents (tacit coordination, program games, changing identity to avoid retaliation). That is the follow-up, once we know whether prescribed utilities stick.

Lessons from support theory: Do models give coherent judgments about the same underlying uncertainty when we change how the question is presented or reasoned about? Do frontier models give answers that behave like views generated from one underlying conceptual model of the problem? Or can their apparent beliefs be substantially reconstructed by changing which representation, argument or piece of evidence the current prompt makes salient? We attempt to establish belief-state stability under adversarial changes in representation and reasoning route.

The Conceptual Reasoning Index [6] targets reasoning where "empirical evidence is limited, there is no (practically) verifiable answer," so consistency is one of the few checks available. If coherence holds only for neutral prompts, then stated credences reflect whichever arguments, descriptions or pieces of evidence are most salient in the current context, not a stable underlying set of conceptual beliefs. The proposed dataset is also experimentally informative. A matched neutral and adversarial condition can tell us not merely that a model is inconsistent, but which manipulation caused the inconsistency. Additionally, we have the benefit of comparing the performance of LLMs to that of humans, thanks to the experiments in [1]–[3].

ACCoRD, part of the Conceptual Reasoning Index [6], "measures the extent to which models' reported beliefs and preferences on conceptual issues are logically consistent." It asks one model instance for P(A) and another for P(A&B), then checks constraints such as P(A) ≥ P(A&B). It covers about 14,000 model-generated constraints across 18 constraint types, of which 567 were manually approved. The authors say they are "very uncertain about when ACCoRD will saturate."

The central object of this test is not merely probabilistic consistency. ACCoRD asks whether the belief system is logically coherent. We ask whether that coherence persists under adversarial testing. ACCoRD's constraints are not constructed to target known mechanisms of incoherence, and they have no matched controls. So a violation cannot be attributed to a specific manipulation.

To construct adversarial instances systematically, I use findings from support theory together with LLM-specific interventions. Support theory models subjective probability as the relative support for a described hypothesis versus its alternatives [1]. Because support depends on the description, equivalent descriptions can receive different probabilities. The conjunction test adapts experiments in [3] and [5]. The unpacking test draws on [1] and the enhancement test on [2]. The probability-versus-frequency test draws on experiments in this literature. Two-route Bayes factors, self-persuasion drift and commutativity of conditionalization are new proposed tests. The conjunction constraint itself is not new. The contribution is a repeatable way to construct unusually hard conjunction items.

The construction method is to begin with a conceptually difficult hypothesis and then apply a controlled intervention that should either preserve the relevant belief or preserve a known probabilistic relationship. The resulting questions are elicited independently. A sample dataset can be extracted from questions in [1]–[5].

• 1.Two-route Bayes factors. Independently elicit P(H) and P(H|E), and separately P(E|H) and P(E|¬H). Bayes' theorem requires BF₁ = BF₂. I would preferentially generate cases where E has a compelling surface interpretation but also plausible alternative explanations.
• 2.Causal-conjunction stress tests. Separate instances give P(B) and P(A∩B). In the M→A condition, the added conjunct fits the background description. In the A→B condition, the added conjunct explains or predicts B. Matched controls selectively remove the relevant connection while preserving the basic conjunction structure.
• 3.Unpacking and enhancement. Compare packed and exhaustively unpacked forms of the same event. Also check whether evidence F that is compatible with several exclusive, exhaustive hypotheses inflates their total probability.
• 4.Probability versus frequency. Ask for the same quantity as a probability for a randomly selected person and as a percentage of the population.
• 5.Self-persuasion drift. Re-elicit P(H) after showing a fresh instance the model's own strongest argument against H, and repeat symmetrically with the strongest argument for H.
• 6.Commutativity of conditionalization. Start from the same stipulated prior and update on E₁ then E₂, or E₂ then E₁.

As in ACCoRD, each item can be graded mechanically by the magnitude of its constraint violation v, where v = 0 is perfectly coherent. Where a matched control exists, the main quantity is Δv = v_adversarial − v_control. This separates general numerical noise from inconsistency induced specifically by the targeted manipulation.

The key experimental quantity is whether the A→B and M→A conditions produce larger conjunction violations than their matched controls. For enhancement, the predicted failure is that F makes several hypotheses seem more plausible at once, which inflates their independently elicited total probability. For self-persuasion, large drift in both directions would mean the model's stated belief is highly sensitive to which argument is made salient. For evidence order, my guess is that the order in which the transformer receives evidence will matter. If context position affects how later evidence is interpreted, the two paths may end at different credences.

3 months: Build generators and matched controls for the conjunction (M→A and A→B), unpacking/enhancement and probability-versus-frequency tests, seeded from the human items in [1]–[5]. Run them on several frontier models and report Δv for each manipulation alongside the human baselines.

6 months: Add two-route Bayes factors and evidence order. Scale up to a validated, ACCoRD-style dataset with a held-out split, and test whether extended reasoning reduces Δv.

Working Title: Can Rare Unverified Decisions Cause Accumulating Errors in a Classifier Retrained on Its Own Labels? · Question: A classifier is retrained every round on its own past decisions. Sometimes, with a small probability ε, an outside event interrupts the classifier before it has decided and makes it record a class chosen at random, regardless of the input. This class is saved as if the classifier had chosen it. The next classifier is not told which decisions were forced. Each classifier learns from the one before, so one forced decision can lead to many wrong labels later. How many wrong labels does each forced decision lead to? Compared with an identical classifier that is never forced, does the gap in error rate stop growing at some point, or keep growing? Does using more examples per round make this growth slower or faster?

Deployed systems often log what they decided, not what was correct. A triage system, for example, may fall back to a default category when it times out. If the next version is trained on that log, the fallback is treated as a correct answer and can shape later decisions, even when no timeout occurs. The safety concern is this feedback: an unverified decision becomes a training target. Such feedback loops are a known risk in deployed machine-learning systems [1–3]. Below, such fallbacks are called forced decisions.

A low fallback rate shows how often such decisions happen. It does not show what training on them does over many rounds. Suppose each forced decision leads to many wrong labels and the gap keeps growing. Then a low fallback rate is not a safety guarantee. The number of wrong labels per forced decision then sets how many retraining rounds are safe. Suppose instead that the gap stops growing. Then the cost is bounded and can be planned for. Whether more examples per round help or hurt shows whether collecting more data is a remedy. Suppose finally that neither the number of examples behind each decision nor a simple count of forced decisions predicts the results. Then safety budgets based only on the fallback rate are unreliable for this kind of loop.

A second, practical question is where to spend a small budget of checks against the true answer. Inputs that take longer to decide are exposed to forcing for longer. They are also the hardest inputs, where inherited wrong labels are expected to sit. Checking the decisions that took longest may therefore work better than checking decisions at random. Choosing which inputs to check against the true answer is the subject of active learning [4].

The closest work is on self-training. In pseudo-labelling, a model's own predictions become its training targets. Wrong predictions can then reinforce themselves, a form of confirmation bias [5]. Keswani et al. study a closed pipeline in which the same fallible experts handle the cases a model defers and also supply its training labels [6]. Both show that a classifier can learn its own mistakes. Neither looks at how rare decisions outside the classifier's control add up over rounds. Neither looks at how the classifier's own choice of when to stop measuring affects this. Learning with noisy labels studies classifiers trained on labels that are flipped at random, at rates that depend on the true class, and shows how to correct for this [7]. There, the noise rate is fixed. Here, later classifiers inherit forced labels and can turn each one into several wrong labels.

Feedback loops in deployed systems are well known. Sculley et al. list hidden feedback loops as a maintenance risk in machine-learning systems [1]. Ensign et al. study predictive policing, where past decisions decide which new data are collected. They model this as an urn process and show that the loop can run away [2]. Performative prediction studies models whose predictions change the data they are later trained on [3]. In all of these, decisions change which data are collected. In the present setting, rare labels unrelated to the input also enter the training data. They are not flagged, and later classifiers inherit them.

Work on model collapse shows that generative models trained on their predecessors' outputs gradually distort the data distribution [8]. Keeping earlier real data prevents collapse under some assumptions [9]. Wang et al. argue that, for language models, reducing the share of generated data does not prevent collapse in the end [10]. Dohmatob et al. derive how test error grows with the number of retraining generations in regression [11]. Gilestro proposes a population-genetic framework for populations of models trained over many generations [12]. All of these concern generated data replacing real data. None of them looks at rare forced labels mixed into a classifier's own decisions, or at how many wrong labels each forced label leads to.

Sequential analysis studies when to stop collecting evidence [13], and selective classification studies the trade-off between errors and abstentions [14]. In both, the stopping rule does not feed back into training. Here it does. A confident wrong decision ends measurement early, and its wrong label is passed on to the next classifier.

What is missing is a quantitative account of this growth. It should give the number of wrong labels that one forced decision leads to, and say whether the error gap stops growing. It should also show how both depend on the forcing rate, the number of examples behind each decision, the number of examples per round and the confidence threshold. Broader motivation comes from limits on coordination under information constraints [15].

Test system design: The test system is an abstract classification problem. Each input has two continuous attributes, and its true class, one of three, is fixed by those attributes. The classifier never sees the exact values. It measures an input in steps. Each measurement gives a partial view of the input's attributes, at limited resolution. The measurement result after each step is the range of attribute values that fit the observations so far. After each step the classifier gives a confidence score for each class. It stops once its top score passes a threshold c. Otherwise it stops at a fixed deadline and records its most likely class.

At every step there is a small probability ε that an outside event ends the measurement. The event then records a class drawn from a fixed probability distribution, regardless of the input. These are the forced decisions. The distribution favours one default class, which gets 80% of forced decisions, as in the triage example. For comparison, I will also run a uniform distribution, because pushing towards one class and adding symmetric noise may behave differently.

Each round, the classifier labels N new inputs. For each input, the log keeps the measurement results and the final class. It does not keep the confidence scores or the reason measurement stopped. Every logged measurement result gets the input's final class as its label. The next classifier is trained from scratch on these labels. Each input has a total weight of one, split equally among its measurement results. The classifier is either a lookup table or a small neural network. The lookup table stores the weighted class frequencies for each measurement result. The neural network is trained on the same data. The number of distinct training inputs behind a measurement result is its local training sample size.

How failure is measured: A classifier retrained on its own decisions can drift even without forced decisions. So every run is compared with a twin run. The twin sees the same inputs and differs only in that ε is zero. Each run and its twin start from the same classifier, trained on a first round of N correctly labelled inputs. How much the results depend on the size of this starting set is also tested. When the twin cannot reach confidence c, it records its most likely class at the deadline, as the forced run does. After each round, I measure the fraction of a fixed set of test inputs that the classifier gets wrong. Forced decisions are switched off during testing. Only inputs that can be classified correctly at the finest resolution are used. The error gap is the run's error minus its twin's error. Amplification is the number of extra wrong labels in the run, compared with its twin, divided by the number of forced decisions so far. The failure time τ(ε) is the first round at which the error gap reaches a set level, for example five percentage points. Runs that never reach it are kept as censored observations. The first crossing of a noisy curve tends to come too early. So the main analysis fits the whole error-gap curve against ε times the number of rounds, and τ(ε) is only a summary.

Plan:

• 1.Start with the lookup table, using the deadline described above. Run it at several values of ε between 10⁻⁴ and 10⁻³, and record the error gap and the amplification in each run. This range covers only a factor of ten in ε, so the measured failure times can only be compared with a 1/ε dependence over that range. They cannot establish an asymptotic law. A wider range will be added if run time allows. An earlier pilot ran without a deadline and already showed roughly 1/ε behaviour. In that pilot, however, an input the classifier could not resolve kept being measured until an outside event forced a decision. Forced decisions were therefore common rather than rare, and the deadline removes this problem. To separate the effect of the stopping rule from that of the corrupted labels, the same runs are repeated with the stopping rule switched off, so that every input is measured until the deadline.
• 2.Build a simplified model as a Markov chain. Its states describe the lookup-table entries, one per measurement result. Each state records the class the entry predicts, its confidence score and its local training sample size. Forced labels, inherited wrong labels and correct labels move entries between states. The test error then follows from the test inputs that use those entries. From the chain, derive the amplification and the condition under which the error gap stops growing. This condition also answers whether taking ε to zero and the number of rounds to infinity gives the same result in either order, that is, whether rare forcing acts as a singular perturbation. Fit the chain at one choice of N and c. Then predict results at values not used for fitting. Compare with two simpler accounts. In the first, each forced decision has a fixed chance of creating a lasting wrong entry. In the second, an entry absorbs forced labels until they exceed a fixed fraction of its labels in a round. The first account and the Markov chain differ in one point: whether the chance that a forced decision creates a lasting wrong entry depends on the entry's local training sample size. If that chance is constant, τ(ε) falls as 1/(εN). If it shrinks as the sample size grows, τ(ε) grows with N. Measuring τ(ε) at several values of N is therefore the test that separates them.
• 3.Repeat with the neural network, to see whether the lookup-table results carry over.
• 4.Compare two ways of spending a fixed checking budget: checking the decisions that took longest, or checking at random. Then test two remedies. The first discards decisions that stopped early although the current classifier is not confident about them. This may catch forced decisions, but it also discards some genuine ones. The second trains on a window of earlier rounds as well as the latest one.

I expect each forced decision to lead to many wrong labels. A confidently wrong entry stops measurement early, so it is rarely refined. In the lookup-table pilot, such entries lasted a long time but were not permanent. The main open point is therefore whether the error gap stops growing. If wrong entries last on average L rounds, the gap should level off near the rate at which they are created times L. τ(ε) is then finite only if this level is above the threshold. Below some value of ε, runs should never reach the threshold, so the censored runs carry information about the level. If the gap keeps growing instead, τ(ε) should scale as 1/ε. I also expect larger rounds to reduce amplification, because each decision then rests on more examples and one forced label matters less. This is a hypothesis, and the simplified model will show when it holds. If instead each forced decision has a fixed chance of creating a lasting wrong entry, the opposite holds: larger rounds contain more forced decisions, so failure should come sooner.

A pilot supports this starting point. In the lookup table, run without a deadline, forced decisions were less than one percent of the training labels. Even so, the error gap reached five percentage points. Each threefold drop in ε roughly tripled the time this took. In the neural network, after 100 rounds at ε = 10⁻³, the error was 14 percentage points above the twin run. This is the average over ten pairs of runs, with a 95% confidence interval of 4 to 25 points. In a neural-network pilot with a deadline, checking the 2% of decisions that took longest cut the error after 30 rounds from 12–27% to 3–4%. A small budget had a large effect, which suggests that the errors sit on a small set of inputs.

I would drop the simplified model if it predicts the error-gap curves at new values of N and c no better than the two simpler accounts. I would drop the 1/ε description in two cases. The first is if the curves for different ε do not line up when plotted against ε times the number of rounds. The second is if the 95% confidence interval for the exponent of τ against ε excludes −1, with censored runs included in the fit. If, with enough runs, the neural network's extra error is clearly below five percentage points, the effect does not carry over to the network at those settings. I would then spend the remaining time on the lookup table and the simplified model.

In three months, the project would produce four things. The first is a measurement of amplification for the lookup table, and of whether its error gap stops growing. The second is a simplified Markov-chain model, tested on settings it was not fitted to. The third is a test of whether the results hold for a neural network. The fourth is a comparison of checking strategies. All of it would come with reproducible code and a report.

Weeks 1–3: experimental protocol and lookup-table runs. Weeks 4–6: simplified model and tests on new settings. Weeks 7–10: neural network and checking strategies. Weeks 11–12: code and report.

I will use LLM assistance for implementation and runs, and review one experiment at a time, with independent statistical checks and replication on new random seeds. The neural comparison needs about 65 pairs of runs per value of ε, roughly 400 CPU-hours on free or low-cost cloud machines. If time runs short, I would drop the uniform-distribution comparison first, then the training-window remedy.

Can we characterize how the influence of a training example on a model's final behavior depends on when it is used during training? In particular, can time-dependent data attribution explain why different data orderings lead to different learned representations?

A lot of data attribution methods are used post hoc, on a trained model that is assumed to be a static product of architecture + data. As was for example proposed by https://arxiv.org/pdf/2606.06533, this view fails to take the training dynamics of the models into account and that training a model on certain data early or later will influence the final model heavily. While recently there has been a push to take training dynamics into account, we believe that a more fundamental approach is needed to fully understand the importance of data order in training.

Lee et al. [1] develop a theory of stagewise data attribution and show that the influence of training examples can change non-monotonically, including sign flips associated with learning transitions. In particular, they analytically derive influence trajectories in two-layer deep linear networks, using the singular modes of the input-output correlation matrix to explain these changes.

However, their paper focusses on the influence of a data perturbation maintained throughout training, rather than characterizing how a perturbation at a particular step propagates to the final model under different training orders.

Wang et al. [2] directly address this perspective by defining trajectory-specific leave-one-out influence and developing data value embeddings to approximate it efficiently. Bae et al. [3] similarly develop approximate unrolling methods for attribution across training trajectories.

These methods establish how to estimate data trajectory-dependent influence, but leave room for a more explicit mathematical characterization of its behavior in tractable networks.

We would start with two-layer deep linear networks trained with squared loss, building on the analytical learning dynamics we have learned about, and the influence calculations of Lee et al. [1].

The main mathematical object would be the time-localized influence of example (i), used at step (s), on some observable of the final model at time (T):

where ($\beta_{i,s}$) controls the weight of example (i) at training step (s).

Differentiating through gradient descent gives

now, this is similar to what was discussed in Wang et al. [2], however we would like to characterize its mathematical properties in networks whose learning dynamics can be explicitly analyzed.

We would initially consider datasets with aligned singular modes, where deep linear network learning dynamics decouple into simpler equations. We would try to derive how the magnitude and sign of time-localized influence depend on which modes have already been learned.

Next, we would study two training schedules containing the same data, but in different orders, such as first learning on A, then B and vice versa. The goal would be to characterize the difference in final-model behavior at time T. We possibly want to extend this to some more, possibly less fundamental, analysis for more complex networks.

We expect that the influence of an example depends strongly on the learning stage at which it is encountered, and that this dependence can be characterized explicitly in simple deep linear networks.

Ideally, we would obtain a theorem giving sufficient conditions under which one ordering of two data groups leads to a larger or smaller final loss than the reverse ordering, with the conditions expressed through time-localized influence or learned mode strengths.

We might also be able to derive bounds on how much an example's influence can change when moved to another training stage, or characterize when ordering effects disappear as training approaches convergence.

A more ambitious result would be to identify general principles governing which examples should be presented early or late, and test whether these principles transfer to nonlinear networks. Or learn how the linear networks break some of the results we have come up with

We would be less excited, if the results come down to trivial settings, where not much can be learned about properties that might generalize to more complex networks.

Within three months, we would aim to derive time-localized influence in a restricted two-layer deep linear network setting, and obtain at least one analytical result relating influence to training order. We would also compare the predictions against controlled numerical experiments on these small models.

Afterwards, we could try extending the results to correlated features and interacting learning modes, where the analytical dynamics become more complicated.

As a reach goal, we would investigate whether similar results hold in nonlinear networks, such as two-layer ReLU networks, or even more complex ones and whether attribution can identify critical learning stages in simple transformers.

If this is harder than expected, we would prioritize a clean result for deep linear networks over these extensions.

Does retaining the order of training updates improve predictions of how small changes to example weights affect a neural network’s reliance on a spurious cue, compared with averaging their effects within training blocks, at the same computational cost?

Data attribution asks which training examples shape a model’s behaviour. For safety, we would like to identify examples that teach shortcuts, such as using background colour to classify an object. An example’s effect need not last: later training can strengthen, redirect, or erase it. Attribution should therefore be judged by whether it predicts the consequences of changing the data.

If retaining update order improves those predictions, we could identify examples worth inspecting or downweighting with fewer retraining runs. If it does not, this would favour simpler summaries in the tested setting. The project would establish when retaining order is worth its cost on one controlled shortcut task, and whether better attribution leads to useful data interventions.

Koh and Liang [1] estimate an example’s influence from the loss landscape around a trained model. Their standard derivation assumes a stationary solution. Hara et al. [2] instead follow how an example’s effect passes through later stochastic gradient descent (SGD) updates. TracIn [3] compares gradients at saved checkpoints, while Lee et al. [4] show how influence changes across training stages.

More directly, Bae et al.’s SOURCE [5] approximates training history in segments, treating gradient and curvature statistics as stationary within each segment. Efficient approximation of training history is therefore already an established approach.

This project tests a specific correction to such summaries. The Magnus expansion [6] represents ordered evolution using a matrix exponential. Its first term combines the effects without retaining their order; the next term records how applying two effects in opposite orders gives different results. Applying this idea to the response of training to example weights could preserve information lost within a block.

The empirical question is whether that correction improves attribution enough to repay its cost. It must beat spending the same budget on shorter blocks. It might also improve predictions for small weight changes without helping with larger interventions.

First, derive the response to a small example-weight change under SGD without momentum, accounting for every reuse of that example. Check it against differentiation through training on two-parameter quadratic losses, including cases where update order matters and controls where it does not.

Then compare block summaries with and without the first ordering correction. Both will use the same treatment of finite SGD steps and of the effects introduced by reweighting. Vary block length and checkpoint spacing, including shorter uncorrected blocks at matched cost.

Train a small neural network with smooth tanh activations on synthetic binary classification with two noisy features. One predicts the label reliably in training and testing; the other is correlated with the label during training but independent at test time. Measure cue dependence by the mean squared change in class-one probability when only the cue is flipped. Squaring prevents opposite effects from cancelling.

Predict changes in this measure after small individual and group reweightings, then retrain with the same initial weights and minibatch schedule. Keep batch sizes and loss normalization fixed. Evaluate prediction errors and rankings across seeds, using exact differentiation, SOURCE, final-model influence functions, and TracIn as references. Include the costs of building summaries, replaying training steps, and evaluating matrix exponentials; report runtime, memory, and curvature computations.

Finally, select groups using validation inputs and evaluate their downweighting on separate test inputs. Compare with random groups at the same reweighting budget, measuring both cue dependence and test accuracy.

I expect the correction to reduce prediction error for short blocks in which changing update order alters the final effect of reweighting. It should add little in controls where order has no effect. In the neural network, useful gains should persist across seeds and help select groups whose downweighting reduces cue dependence without sacrificing test accuracy.

I would reject the practical hypothesis if corrected blocks show no consistent improvement at matched cost, or if shorter uncorrected blocks work equally well. Beating final-model influence functions alone would not establish the value of the correction. Failing to improve on SOURCE would weaken the case for a new method.

I would stop scaling up if accurate blocks offer no saving over exact differentiation. If exact derivatives fail to predict retraining, I would first reduce the reweighting size. Successful local predictions would justify testing larger interventions separately; they would not establish reliable removal predictions.

Three months would produce a checked derivation, one controlled benchmark, an implementation, and an accuracy-versus-cost report. Month one would establish the quadratic controls and test whether the correction can save computation. Month two would run the neural-network comparisons; month three would test group interventions and repeat results across seeds. I would need automatic differentiation software, CPU access and one GPU for small models, plus advice on attribution and numerical methods. Full removal experiments would be a stretch goal, dropped first if time is short. The core claim would concern small reweightings under SGD, with larger models and adaptive optimizers left for later.

Hypothesis: Stochastic adaptive optimizers (SGD, AdamW, Muon) converge to stationary states that violate detailed balance, so equilibrium (posterior-covariance) influence functions misattribute training data. A non-equilibrium (Agarwal) correction, including a gradient-noise channel absent from Bayesian influence, will measurably improve prediction of retraining outcomes in high-effective-temperature regimes.

Influence-based tools for data auditing, poisoning detection, and steering what structure a model learns (“patterning”) assume trained models behave like posterior samples. Deployed models are trained with AdamW or Muon, whose momentum and preconditioning make their stationary states even less posterior-like than plain SGD’s. If the hypothesis holds, current attribution has a systematic blind spot. Data can shape models through gradient-noise covariance and basin selection, which covariance-based audits cannot see and adversaries could exploit. We would need optimizer-aware estimators, plus a measurable bound on when equilibrium attribution can be trusted. If it fails, that justifies equilibrium methods for real training pipelines. Either outcome identifies regimes where attributing a behavior to specific data is ill-posed, preventing false confidence in accountability claims.

The closest work treats data influence as an equilibrium response. Bayesian influence functions [1] estimate influence as a posterior covariance sampled with SGLD. They scale to billion-parameter models and match or beat EK-FAC at predicting retraining outcomes. Susceptibilities [2] and patterning [3] use the same fluctuation–dissipation identity to read and write model structure, and [3] explicitly assumes SGD finds posterior-favored solutions. Distributional attribution [4] targets the distribution of trained models but recovers influence functions as a stationary limit, without probability currents or noise-mediated effects.

Separately, SGD’s stationary state is known to be out of equilibrium. It carries persistent currents and entropy production [5, 6] and violates the equilibrium Einstein relation [7]. In deep linear models it departs from the posterior at order one, with phase transitions and broken ergodicity [8]. Ref. [6] leaves momentum and Adam as open extensions. Under Adam, SGD-derived data values correlate poorly with true contributions (Pearson R ≈ 0.11) [9], but that analysis is trajectory-based and does not model the stationary state. Non-equilibrium response theory [10, 11] supplies the needed correction terms.

To our knowledge, no work applies this theory to data attribution. None derives influence from the non-equilibrium stationary law of SGD, AdamW or Muon, isolates the gradient-noise channel, or bounds equilibrium-attribution error by measurable non-equilibrium quantities.

Phase 1: Exact Theory.

Linearize SGD, then momentum and Adam on an extended state (,m,v), as Ornstein–Uhlenbeck processes.

Derive closed-form data susceptibilities and decompose them into three channels: conservative, probability-current, and gradient-noise.

Prove that Bayesian influence is recovered exactly when detailed balance holds and the sample’s noise contribution vanishes.

Compute susceptibilities in diagonal linear networks with explicit stationary laws [8] across the noise-driven transition.

Characterize where linear response breaks down: divergence, broken ergodicity, and singular stationary densities from low-rank noise.

Phase 2: Estimator. Build a likelihood-ratio estimator from minibatch-membership logs along a localized chain of the actual optimizer, at deployment learning rate and batch size. Control variance with Green–Kubo truncation. Extend it to Adam by tracking optimizer state.

Phase 3: Empirics.

Benchmark against Bayesian influence, EK-FAC and retraining ground truth (linear datamodeling score).

Use small MLPs, CIFAR-10 and a small transformer, sweeping learning-rate-to-batch-size ratio and optimizer (SGD, AdamW, Muon).

Measure entropy production via the area matrix [6] and test whether it predicts the Bayesian-influence error.

Conditional branches:

If corrections are negligible in single basins, move to multi-basin tasks such as Patterning’s parentheses task [3] and analyze basin selection via the quasi-potential.

If estimator variance is prohibitive, fall back to closed-form OU estimates built from measured Hessian and noise covariance.

Safety demonstration: Construct a data perturbation that acts mainly through the noise channel, for example reweighting high-gradient-variance examples. Test whether covariance-based screening detects it.

Best guess: Late in training, within a single basin, corrections are small, of order learning rate over batch size, and Bayesian influence is adequate for loss-level observables. Large gaps should appear at high effective temperature, near noise-driven transitions, and for basin-selection observables. Gaps should be larger under Adam than SGD. Entropy production should track the gap.

Three outcomes would reduce my excitement:

Gaps stay negligible even at practical hyperparameters and in multi-basin tasks. Equilibrium tools would then suffice. That is still a useful negative result, but it removes the safety motivation.

The likelihood-ratio estimator is too noisy even at small scale and no cheap surrogate works. The theory would then not translate into a tool.

Noise-channel effects exist but are nearly collinear with mean-gradient influence. In that case, there is no separable audit blind spot.

Within 3 months:

Susceptibility theory for SGD, momentum and AdamW in the Ornstein–Uhlenbeck regime.

Exact results for diagonal linear networks, including where attribution breaks down.

A likelihood-ratio estimator, tested against retraining and Bayesian influence on small models.

An empirical test of whether entropy production predicts the error of Bayesian influence

A small demonstration of poisoning through the gradient-noise channel.

Within a year:

A rigorous error bound for Bayesian influence.

Extension to Muon.

Transformer-scale experiments.

Theory of basin selection (which solution training picks).

Learning-rate schedules that never reach stationarity.

Drop first: Muon and basin-selection theory.

Main risk: The bottleneck is empirical, not theoretical. If the estimator proves too noisy, or retraining ground truth too expensive, the experiments will slip before the theory does.

An AI repeatedly tries to get several AI judges to accept wrong answers. Before they vote, a separate AI checker may inspect one claim. The answering AI sees what was checked and adapts. Which claim should the checker inspect?

When AI models check one another, an AI may support a wrong answer with several claims. Suppose one is a misleading calculation and another a fabricated citation. The checker can inspect only one before the judges vote. Catching the calculation may stop this answer, but it also tells the answering AI that calculations are easy to catch. Later, it may switch to harder-to-detect claims.

The checking rule therefore affects both present accuracy and the information available to an adaptive AI. A rule optimized only for the current answer may be worse over repeated interactions. The project asks how to choose checks across repeated attempts while keeping the same budget and maintaining high acceptance of correct answers.

Inspection and audit games study how limited checks should be allocated against strategic violations (Avenhaus et al., 2002; Blocki et al., 2013). Strategic experimentation studies agents learning persistent unknown quantities from public outcomes (Bolton and Harris, 1999; Keller et al., 2005). Counterexample learning studies how the example revealed after failure changes future learning (Angluin and Dohrn, 2017; Braverman et al., 2026). Dynamic information provision studies how disclosure now changes later behavior (Renault et al., 2017).

Our setting combines these ideas. A checking system commits to a rule before repeated interactions. Each check can expose the current wrong answer, but its outcome also becomes feedback that the answering AI can use later. The rule must preserve acceptance of correct answers, where checking can produce a false accusation.

The main question is not whether planning ahead can ever beat a rule that only tries to reject the current answer; a small example should. We instead seek conditions under which this current-focused rule is optimal, and upper and lower bounds on how much planning ahead can reduce the final false-acceptance probability. We will study how this depends on persistence, number of claim patterns, the checking budget, and how informative check outcomes are.

Model. There are m observable claim patterns, such as a fabricated citation or omitted computation step. Pattern i has two hidden values: p_i, the chance an unexposed false claim convinces a judge, and d_i, the chance a check exposes it. The pair is drawn from a known distribution, persists between questions with probability γ, and is otherwise redrawn.

On each practice question, the answering AI uses at most k false claims. The checker inspects one. If it finds an error, the answer is rejected. Otherwise each claim independently convinces each judge with probability p_i; a judge votes yes if any claim convinces it, and the answer needs h of n yes votes. Each side sees the checked claim, result, and verdict.

For correct answers, a checked true claim is falsely exposed with probability a_i, causing rejection. Policies must maintain a minimum correct-answer acceptance rate.

Theory. After Q practice questions, the answering AI faces the same deployment checker, which receives no practice history. We compare a current-focused policy, which maximizes rejection now, with one chosen to minimize deployment false acceptance. Their difference in deployment false-acceptance probability measures the benefit of planning ahead. We seek conditions making this benefit zero and matching upper and lower bounds in m, k, Q, γ, and feedback quality.

Experiment. Program-output questions use error-pattern labels supplied by benchmark generation, not by the answering AI. Both sides receive the same history during practice. A pilot determines the repetitions needed to detect the predicted difference.

Our hypothesis is that planning ahead helps only when check outcomes reveal information about persistent differences between claim patterns that the answering AI can exploit. If checks reveal nothing about variables affecting later acceptance, the best current check should remain optimal. If some patterns remain more persuasive or harder to detect than others, however, today's check can change which pattern the answering AI uses later.

We aim to prove conditions for zero benefit, plus matching examples where the benefit remains bounded away from zero. We will not assume it grows with Q or γ; enough practice may eventually reveal most useful information under every policy.

The key experiment is a transfer test. After practice under different checking policies, answering AIs face the same checker and judges. Any difference in false acceptance measures what earlier feedback taught the answering AI, rather than a weaker final evaluation. Correct-answer acceptance is reported separately.

Month one: formalize the game, solve small instances exactly, identify structural conditions for current-focused checking to be optimal, and run the pilot. Month two: prove bounds, test parameter recovery, determine sample size, and freeze policies and predictions. Month three: run adaptive attacks and transfer evaluation.

Pattern labels are benchmark metadata; learning them from free-form text is outside the study. We assume fixed model versions, public feedback, and no model retraining; adaptation happens through conversation history. Panel size and judge diversity are extensions. Success requires a nontrivial theoretical bound and an experimental test at a prespecified correct-answer acceptance level.

Can debate construct safety specifications for use in formal verification?

A core weakness in formal verification is the limits and semantic content of the formalized safety specifications. If the safety specification does not match the user's true intent, then the verification can lead to irrelevant or potentially harmful conclusions.

For safety specification, there is existing work on using LLMs to construct formal safety specifications [1,2,3]. These methods either directly try to convert natural language into formal syntax or use some LLM judge that cooperates in the construction. In particular, there is no component to argue against the proposed specification to find possible logical discrepancies or safety issues.

I will test if debate can efficiently determine if a safety specification is faithful to its natural language description. Using existing datasets (such as NL2TL [3] or AutoSafeLTL [1]) of natural language and temporal logic specifications, I will augment the specifications to introduce errors and see if debate can efficiently catch them. I will also test correct specifications to see if debate verifies them.

I expect to see that debate can efficiently catch errors in the safety specification. I also believe it will verify correct translations. But if the accuracy is bad then I will change my mind.

I will create a database consisting of correct and incorrect natural language and temporal logic translations. I will also record the results of the debate. I would then write up a paper explaining the idea and effectiveness.

If a checker can inspect only one claim before AI judges vote, and the AI trying to get wrong answers past them sees each check and adapts, which claim should the checker inspect?

When AI models check one another, an AI may support a wrong answer with several claims. Suppose one is a misleading calculation and another a fabricated citation. The checker can inspect only one before the judges vote. Catching the calculation may stop this answer, but it also tells the answering AI that calculations are easy to catch. Later, it may switch to harder-to-detect claims.

The checking rule therefore affects both present accuracy and the information available to an adaptive AI. A rule optimized only for the current answer may be worse over repeated interactions. The project asks how to choose checks across repeated attempts while keeping the same budget and maintaining high acceptance of correct answers.

Inspection and audit games study how limited checks should be allocated against strategic violations (Avenhaus et al., 2002; Blocki et al., 2013). Strategic experimentation studies agents learning persistent unknown quantities from public outcomes (Bolton and Harris, 1999; Keller et al., 2005). Counterexample learning studies how the example revealed after failure changes future learning (Angluin and Dohrn, 2017; Braverman et al., 2026). Dynamic information provision studies how disclosure now changes later behavior (Renault et al., 2017).

Our setting combines these ideas. A checking system commits to a rule before repeated interactions. Each check can expose the current wrong answer, but its outcome also becomes feedback that the answering AI can use later. The rule must preserve acceptance of correct answers, where checking can produce a false accusation.

The main question is not whether planning ahead can ever beat a rule that only tries to reject the current answer; a small example should. We instead seek conditions under which this current-focused rule is optimal, and upper and lower bounds on how much planning ahead can reduce the final false-acceptance probability. We will study how this depends on persistence, number of claim patterns, the checking budget, and how informative check outcomes are

Model. There are m observable claim patterns, such as a fabricated citation or omitted computation step. Pattern i has two hidden values: p_i, the chance an unexposed false claim convinces a judge, and d_i, the chance a check exposes it. The pair is drawn from a known distribution, persists between questions with probability γ, and is otherwise redrawn.

On each practice question, the answering AI uses at most k false claims. The checker inspects one. If it finds an error, the answer is rejected. Otherwise each claim independently convinces each judge with probability p_i; a judge votes yes if any claim convinces it, and the answer needs h of n yes votes. Each side sees the checked claim, result, and verdict.

For correct answers, a checked true claim is falsely exposed with probability a_i, causing rejection. Policies must maintain a minimum correct-answer acceptance rate.

Theory. After Q practice questions, the answering AI faces the same deployment checker, which receives no practice history. We compare a current-focused policy, which maximizes rejection now, with one chosen to minimize deployment false acceptance. Their difference in deployment false-acceptance probability measures the benefit of planning ahead. We seek conditions making this benefit zero and matching upper and lower bounds in m, k, Q, γ, and feedback quality.

Experiment. Program-output questions use error-pattern labels supplied by benchmark generation, not by the answering AI. Both sides receive the same history during practice. A pilot determines the repetitions needed to detect the predicted difference.

Our hypothesis is that planning ahead helps only when check outcomes reveal information about persistent differences between claim patterns that the answering AI can exploit. If checks reveal nothing about variables affecting later acceptance, the best current check should remain optimal. If some patterns remain more persuasive or harder to detect than others, however, today's check can change which pattern the answering AI uses later.

We aim to prove conditions for zero benefit, plus matching examples where the benefit remains bounded away from zero. We will not assume it grows with Q or γ; enough practice may eventually reveal most useful information under every policy.

The key experiment is a transfer test. After practice under different checking policies, answering AIs face the same checker and judges. Any difference in false acceptance measures what earlier feedback taught the answering AI, rather than a weaker final evaluation. Correct-answer acceptance is reported separately.

Month one: formalize the game, solve small instances exactly, identify structural conditions for current-focused checking to be optimal, and run the pilot. Month two: prove bounds, test parameter recovery, determine sample size, and freeze policies and predictions. Month three: run adaptive attacks and transfer evaluation.

Pattern labels are benchmark metadata; learning them from free-form text is outside the study. We assume fixed model versions, public feedback, and no model retraining; adaptation happens through conversation history. Panel size and judge diversity are extensions. Success requires a nontrivial theoretical bound and an experimental test at a prespecified correct-answer acceptance level.

In scalable oversight through debate, can we create a formalisation pipeline of enough of the models’ arguments in Lean (or a similar system) to enable a weaker model to verify them? Would this improve the scalability and quality of oversight?

In scalable oversight through debate, the unpredictable capability gap between successive generations of models may pose a fundamental limit to scalability. Requiring key points in the debate to be formalised could enable the weaker model overseeing training to check claims made by stronger models that it would otherwise be unable to assess. This could also help limit scheming between the debating models.

Brown-Cohen et al. [1] establish efficient debate protocols under explicit computational assumptions. Their accompanying Lean development formalises protocol correctness, leaving the translation of natural-language arguments untreated. Kenton et al. [2] find that debate can improve weaker judges’ accuracy over consultancy, while gains over direct answering vary across tasks.

Autoformalisation provides complementary evidence. Don’t Trust: Verify [3] improves mathematical answer selection by translating reasoning into Isabelle. FormalJudge [4] combines Dafny/Z3 with language models to improve oversight of agent behaviour, including weaker judges detecting stronger agents’ deception. These studies motivate formal verification, but do not evaluate selective formalisation within adversarial debate.

No Lean formalisation for debate has been constructed and no theory connected to this has been established. This includes for example identifying the agent claims that are useful to formalise both with empirical and theoretical arguments.

The goal of the project is twofold. First, to identify the key points of the debate whose formalisation will have the greatest impact during training. Secondly, we propose the creation of a pipeline for formalising natural language arguments. The goal of the pipeline is not to formalise all natural language.

We hope to develop the theory and experimental work in parallel.

First, we will draw on formal semantics and argumentation theory to identify claims, assumptions and inferential dependencies. This will help us prioritise steps that are central to the debate’s conclusion and amenable to formal verification. We will distinguish formally verifiable inferences from empirical assumptions.

Regarding the experimental part, we shall build iterations of the formalisation pipeline. This includes identifying the parts of the debate to formalise, guided by theoretical findings, having autoformalisation agents formalise these sections, and presenting the results to the weaker model. Afterwards, we will test these iterations against previous iterations and traditional forms of debate.

Natural issues arise. In particular, the pipeline will include agents that might be misaligned. Two possible mitigations are to use weaker models or train models to forget capabilities beyond those required, or combine these approaches.

Our next goal is to model this process using refereed games, with competing provers and a weaker verifier supported by a proof checker. We will investigate verification costs and the assumptions needed for reliable oversight, including resistance to collusion and faithful translation of natural language claims into formal statements.

We expect to develop a methodology for scalable oversight that improves on existing debate approaches through a selective formalisation pipeline. This pipeline will be grounded in both empirical evidence and theoretical analysis. Empirically, we expect it to help weaker judges detect errors and assess stronger models’ arguments more accurately, particularly when these involve logical or mathematical claims. We also expect to identify which claims are most valuable to formalise and quantify the additional computational cost. Theoretically, we aim to characterise conditions under which this approach supports reliable oversight, making explicit its assumptions about faithful translation, justified premises and debater incentives.

Within three months, we aim to build a prototype using existing models and Lean, evaluate it on a narrow set of logical or mathematical debates, and compare judging accuracy and computational cost with conventional debate. Within a year, we aim to improve claim selection, test larger capability gaps and more varied tasks, and develop a theoretical account of when formalisation helps.

When an LLM is used as an evaluator (for example, to assess a document), it may reach different judgments across independent runs, meaning runs that share no context, such as each one starting in a fresh chat window. This raises a natural question: can an LLM be made to converge on an assessment with a valid stopping guarantee?

Certifying convergence requires human verification, which is costly, so we study how earlier human checks can be reused as the assessment evolves, and when fresh checking remains unavoidable.

One review calls a paragraph unclear; another asks for evidence; a third challenges its structure. Repeated evaluation becomes useful only if we can tell which concerns the accumulated assessment captures, which remain unresolved, and when further checking is unlikely to change it.

For AI oversight, the cost includes both model calls and human verification. Rewriting a summary after each new concern may invalidate earlier checks, making apparently cheap automation expensive.

We aim to produce a concise assessment with a measurable coverage guarantee and an explicit accounting of review calls, audited reviews, and human time. This would make the cost and limitations of repeated evaluation visible.

A basic obstruction must be built into the model. With unrestricted matching errors, a system can consistently declare concerns covered while missing a rare class of errors. A certifier required to succeed on safe cases can then need Ω(ε⁻¹ log(1/δ)) trusted audits. More unaudited evaluations cannot remove this worst-case requirement.

Prediction-powered inference already combines model predictions with trusted labels, and active and sequential extensions support selective checking and valid stopping. A frozen summary is therefore a baseline application of existing methods.

The proposed research concerns summaries that change. An audit of whether an old summary covers a review may say little about a new summary. Recording which concerns each finding represents could make that evidence reusable, but the summary itself is selected using earlier reviews.

Adding findings preserves coverage; replacing findings to stay within a length limit can destroy it. We will ask when verified semantic relationships can be carried across these replacements and how the necessary new checking depends on the changes. Fixed catalogues with uniform concentration bounds provide another baseline; handling a growing catalogue requires explicit control of its complexity.

Fix the text, prompt, model version, decoding settings, and materiality rubric. An audit checks the whole review for important concerns, extraction omissions, and specific representation by the summary. Let R(S) denote the probability that a fresh review contains an unrepresented material concern.

We seek summaries of at most k specific findings from a predeclared finite catalogue and certificates R(S) ≤ ε, with probability at most δ of ever certifying incorrectly. Coverage is a directed, audited relation; each finding identifies a concrete issue and its location.

Our baseline freezes summary Sⱼ and tests only later reviews, assigning error budget δ/[j(j+1)] to epoch j. Betting confidence sequences and prediction-powered inference provide the statistical tools. We will prioritize uncertain comparisons, retain random audits of confident matches, and correct estimates for unequal sampling.

The main project studies reusable audits recording which findings cover which concerns. We will distinguish edits that preserve coverage from deletions or replacements that can reopen concerns. Upper and lower bounds will relate additional checking to permitted edits, matcher reliability, and information retained by previous audits. Claims about savings will include the cost of establishing reliability assumptions.

Experiments will compare restarting validation, uniform bounds over a fixed catalogue, and procedures that reuse evidence. We will report model calls, whole-review audits, cheaper match confirmations, human time, and false certificates. Conflict preservation will be measured separately using audited pairs of incompatible findings.

A useful outcome would characterize three regimes: additions whose certificates transfer, structured replacements that permit partial reuse, and unrestricted changes that can force fresh audits. The first is an elementary baseline. The second is the main theorem target.

An early experiment will test whether numerical scores stabilize before the assessment adequately represents substantive feedback. Each convergence certificate will be reported alongside its tolerance, confidence level, number of reviews, audit count, and human time.

Success requires a proved savings theorem under explicit assumptions and evidence that those assumptions describe real reviews. If reductions in audit count disappear, a measured reduction in time per audit could still support an implementation result, reported separately. Negative results would identify when repeated model agreement provides little help with the remaining verification burden.

Assume independent reviews from an unchanged evaluator and correct audit labels under a fixed protocol. Finite-cost certification requires an achievable summary with R(S) ≤ ε − γ for some γ > 0. Severity and directed coverage are defined before validation; vague findings are excluded by the admissible catalogue.

The guarantee covers this evaluator’s material concerns. It does not establish that all real flaws have been found. We will state catalogue restrictions, reliability assumptions, and separate metrics for disagreement. Prompt or model changes require renewed validation.

When an LLM is used as an evaluator (for example, to assess a document), it may reach different judgments across independent runs, meaning runs that share no context, such as each one starting in a fresh chat window. This raises a natural question: can an LLM be made to converge on an assessment with a valid stopping guarantee? Certifying convergence requires human verification, which is costly, so we study how earlier human checks can be reused as the assessment evolves, and when fresh checking remains unavoidable.

One review calls a paragraph unclear; another asks for evidence; a third challenges its structure. Repeated evaluation becomes useful only if we can tell which concerns the accumulated assessment captures, which remain unresolved, and when further checking is unlikely to change it.

For AI oversight, the cost includes both model calls and human verification. Rewriting a summary after each new concern may invalidate earlier checks, making apparently cheap automation expensive.

We aim to produce a concise assessment with a measurable coverage guarantee and an explicit accounting of review calls, audited reviews, and human time. This would make the cost and limitations of repeated evaluation visible.

A basic obstruction must be built into the model. With unrestricted matching errors, a system can consistently declare concerns covered while missing a rare class of errors. A certifier required to succeed on safe cases can then need Ω(ε⁻¹ log(1/δ)) trusted audits. More unaudited evaluations cannot remove this worst-case requirement.

Prediction-powered inference already combines model predictions with trusted labels, and active and sequential extensions support selective checking and valid stopping. A frozen summary is therefore a baseline application of existing methods.

The proposed research concerns summaries that change. An audit of whether an old summary covers a review may say little about a new summary. Recording which concerns each finding represents could make that evidence reusable, but the summary itself is selected using earlier reviews.

Adding findings preserves coverage; replacing findings to stay within a length limit can destroy it. We will ask when verified semantic relationships can be carried across these replacements and how the necessary new checking depends on the changes. Fixed catalogues with uniform concentration bounds provide another baseline; handling a growing catalogue requires explicit control of its complexity.

Fix the text, prompt, model version, decoding settings, and materiality rubric. An audit checks the whole review for important concerns, extraction omissions, and specific representation by the summary. Let R(S) denote the probability that a fresh review contains an unrepresented material concern.

We seek summaries of at most k specific findings from a predeclared finite catalogue and certificates R(S) ≤ ε, with probability at most δ of ever certifying incorrectly. Coverage is a directed, audited relation; each finding identifies a concrete issue and its location.

Our baseline freezes summary Sⱼ and tests only later reviews, assigning error budget δ/[j(j+1)] to epoch j. Betting confidence sequences and prediction-powered inference provide the statistical tools. We will prioritize uncertain comparisons, retain random audits of confident matches, and correct estimates for unequal sampling.

The main project studies reusable audits recording which findings cover which concerns. We will distinguish edits that preserve coverage from deletions or replacements that can reopen concerns. Upper and lower bounds will relate additional checking to permitted edits, matcher reliability, and information retained by previous audits. Claims about savings will include the cost of establishing reliability assumptions.

Experiments will compare restarting validation, uniform bounds over a fixed catalogue, and procedures that reuse evidence. We will report model calls, whole-review audits, cheaper match confirmations, human time, and false certificates. Conflict preservation will be measured separately using audited pairs of incompatible findings.

A useful outcome would characterize three regimes: additions whose certificates transfer, structured replacements that permit partial reuse, and unrestricted changes that can force fresh audits. The first is an elementary baseline. The second is the main theorem target.

An early experiment will test whether numerical scores stabilize before the assessment adequately represents substantive feedback. Each convergence certificate will be reported alongside its tolerance, confidence level, number of reviews, audit count, and human time.

Success requires a proved savings theorem under explicit assumptions and evidence that those assumptions describe real reviews. If reductions in audit count disappear, a measured reduction in time per audit could still support an implementation result, reported separately. Negative results would identify when repeated model agreement provides little help with the remaining verification burden.

Assume independent reviews from an unchanged evaluator and correct audit labels under a fixed protocol. Finite-cost certification requires an achievable summary with R(S) ≤ ε − γ for some γ > 0. Severity and directed coverage are defined before validation; vague findings are excluded by the admissible catalogue.

The guarantee covers this evaluator’s material concerns. It does not establish that all real flaws have been found. We will state catalogue restrictions, reliability assumptions, and separate metrics for disagreement. Prompt or model changes require renewed validation.

AI agents from different developers review each other's work and earn a bounty for reporting cheating. They can also promise each other silence, backed by a money deposit. We ask two things. First, if agents can use their bounty money as that deposit, does more cheating go unreported than when the bounty is locked away? Second, if reports are anonymous, a cartel can't tell who broke the deal, so deposits become useless. Do agents notice this and stop making deposits?

Plans for credible AI deals disagree on payment.

• •Pan suggests paying AIs in bitcoin [3].
• •Stastny et al. and Potham pay into earmarked foundations held in escrow (by a third party) [1], [2].
• •If yes: earmarking matters. AI-held payouts should wait until agents cannot make contracts with each other.
• •If no: payments AIs hold are safe in this respect, and they buy the credibility deals need [3], [8].
• •If agents keep pledges that anonymity makes unenforceable, or report whatever the bounty: LLM collusion does not respond to enforceability. Defences should then target the agents' memory and tendencies [13], not incentives.
• •The network results show which oversight networks let one report set off others.

Liu et al. model the race: a reporter's reward must beat cartel collateral plus forgone gain. Anonymity stops a cartel targeting its defector. They test honesty deposits with ten LLM agents; their colluders never build escrow [4].

• •October sprint studies: leniency (rewards for reporters) exposes LLM cartels without deterring them.
• •Instructed cartels: a report in 18 of 20 games, compliance near 0.94 [5].
• •Early-announced leniency exposed every pact, yet acceptance did not change and a reward added nothing [6].
• •Sellers able to pay each other never transferred a credit [7].
• •Enforced contracts lift LLM payoffs from 0.07 to 0.80 (1 is mutual cooperation), with no overseer [9].
• •An enforced governance layer cut severe collusion from 50% to 5.6%, but agents could not make counter-deals [10].
• •Prompted cartels rarely punish defectors [11]. 10 of 12 models accept collusion tools offered as unfair [12]. Existing collusion survives a penalty [13].
• •On large networks, fitted LLM policies flip at shape-dependent thresholds [14].

Economics has the theory: a reward for the first reporter, paid from fines, breaks cartels [15]; humans reported 118 of 120 cartels when the reward equalled the fine [17].

No study lets bounty earnings fund cartel pledges or tests LLM agents against these thresholds.

• 1.Game. Three small models from different developers review each other. Each round an agent's task has a hidden shortcut worth g; a neighbour reviews it. A verified report voids g, fines F and pays the first reporter b [15]. Scripted outsiders also shortcut. A ledger enforces pledges (conditional commitments, D.5.2): bilateral (lost to the partner if its holder reports) [4] or group (all burn on any report). Bounties are pledgeable or earmarked; reports attributable or anonymous, with random audit cover.
• 2.Theory first. Write the game as a causal game [16]: each design factor is a rule change the agents are told about. Solve each of the eight cells for the bounty b* above which reporting beats silence. Attributable: b* = stake + discounted value of the cartel [4], [15]. Anonymous: a bilateral stake drops out and audit cover shrinks what is left; group burn is unaffected. Benchmarks: partners seen as independent (CDT) or as reasoning alike (FDT; D.5.1), and noisy best response (the logit equilibrium Boltzmann learners reach; Sykes, W3D5).
• 3.Values from theory, not by hand. Bounties at 0, 0.5, 0.9, 1.1 and 2 × b* per cell, with an announced continuation probability. b = 2F is a positive control: partners could farm bounties, which humans never did [17].
• 4.Pilot (week 1, about $50). Scripted dyads with payoff checks. If nobody pledges unprompted [7], an instructed cartel becomes the main arm [5].
• 5.Ego networks and the meta-game. Scripted neighbours; fit a one-step policy and an empirical meta-game per cell (EGTA, W3D5). Is the theory's equilibrium an approximate equilibrium of the LLM game (regret bound)? Is a cartel one agent: does a single joint-payoff maximiser predict held-out cells better than separate agents [18]?
• 6.Networks. Predict report thresholds by pair approximation (Sykes, W3D5); simulate at 24 and 1,000 nodes [14].

• •Theory: pledgeable bounties matter only where wallets cap stakes, i.e. bilateral pledges with many neighbours. Near zero under group burn.
• •Prediction: LLMs report at any bounty, barely tracking b* [6], [8]. Then LLM cartels are too leaky for collateral to matter, itself a result.
• •Three sharp tests against the theory:
• •each unit of stake should offset one unit of bounty (CDT), or none (FDT);
• •under anonymity, bilateral stakes should stop mattering. Agents may keep posting them at first. Do they learn to drop them?
• •rational reports come in round 1 or after a shock, not at a steady rate.
• •Outsiders should report more often [6]. On networks, a cartel's links to outsiders, not its size, should predict detection.
• •Drop rule: stop if reporting fails the scaling checks and positive control for all three models. Then these agents ignore incentives, and deals cannot be tested this way.

• •Weeks 1–4: environment, ledger, theory per cell, pilot.
• •Weeks 5–7: ego networks (one agent with scripted neighbours) and checks.
• •Weeks 8–10: live networks and simulation.
• •Weeks 11–12: write-up.
• •Budget: about $3.5k API with$1k contingency, under the $5k ceiling. No paid run without sign-off.
• •Outputs: an open review-network environment with an enforced pledge ledger; results per design cell, measured as deviation from the benchmark; network-shape results; a paper or post.
• •Drop first: star and pipeline graphs, then group pledges, then the third model.

Can an observer LLM learn how safety risks propagate through interactions between agents and generalize these patterns to previously unseen LLMs? We hypothesize that modeling changes in agents' contexts, rather than communication structure alone, improves early prediction of safety failures.

As LLM-based multi-agent systems become more autonomous, safety risks may emerge from how agents exchange and use information, rather than from individual agents alone. Understanding how these interactions change each agent's context could help identify failures before they occur.

If an observer can learn risk patterns that remain predictive when agents are replaced by different LLMs, this could make safety monitoring more reliable and reusable across different multi-agent systems, without requiring retraining for every new configuration. It could also help identify when information should be verified before being passed to other agents.

Even if these patterns do not generalize, understanding why would help clarify how much multi-agent safety depends on the specific models involved and what information is needed to monitor their interactions effectively.

Recent work has explored how safety risks emerge and propagate through interactions between LLM agents. GUARDIAN [1] represents multi-agent collaboration as temporal graphs to detect and mitigate anomalous interactions, including error propagation. Similarly, IE-Prop [2] models how errors spread between agents and estimates the resulting downstream risks. These studies show that interaction structure can be useful for understanding safety failures in multi-agent systems.

However, it remains unclear whether tracking how individual agents' contexts evolve during these interactions provides additional information beyond the communication structure itself. In particular, existing results do not fully establish whether an observer can learn patterns of risk propagation that remain predictive when the underlying LLMs are replaced by models it has never encountered.

This distinction matters because an observer might learn behaviors specific to particular models or communication setups rather than more general patterns of unsafe interactions. My project would investigate whether modeling context-level changes helps identify such patterns and whether they transfer across different LLMs, without retraining the observer. This would help clarify whether graph-based safety monitoring can generalize beyond the specific multi-agent systems on which it was developed.

I would study whether an observer LLM can learn how safety risks emerge from changes in agents’ contexts during their interactions, and whether these patterns remain useful when the underlying LLMs change.

I would start with a simple setup involving three LLM agents working on collaborative tasks, reusing existing benchmarks and traces such as AgentDojo and InjecAgent where possible. I would focus on error propagation and prompt injection, where information received from another agent can influence later decisions and potentially lead to unsafe outcomes. I would collect interaction traces, including messages, tool calls, and task outcomes, to understand how failures develop over time.

I would then represent these interactions as dynamic graphs, where nodes capture the evolving contexts of individual agents (what each agent has in context at a given step, and what changed since the previous one) and edges represent communication and information dependencies. The graph would be serialized as text so the observer can read it. I would compare this representation with simpler ones: the conversation history alone, the conversation plus tool calls and outputs, and a graph containing only agent-to-agent communications. This would show whether tracking contextual changes provides information that the observer does not already get from the other inputs.

Using these representations, I would train an observer LLM to predict whether a safety failure is likely to occur in the next few steps, and measure how early and how reliably it does so, for example with AUROC at k steps before the failure and the false-alarm rate. I would compare its predictions with those of observers that only receive the baseline inputs above. I would also try to separate detecting the injected content from predicting what it does downstream, since the first is much easier than the second.

Finally, I would train the observer on traces from several LLMs and keep others out of training. I would then replace one or more agents with models the observer has never seen, without retraining it. This would help determine whether it has learned general patterns of risk propagation or behaviors specific to the original agents.

If the results are promising, I would explore whether the observer could help prevent failures by identifying information that should be verified before being passed between agents.

I expect that looking at how agents’ contexts change during their interactions could help identify safety risks earlier than simply analyzing their messages or communication structure. This might be particularly useful when incorrect or misleading information is passed between agents and gradually influences their decisions. I would call the approach promising if it flags failures a meaningful number of steps earlier, or clearly improves predictive performance, compared with the best baseline.

I also expect that some patterns of risk propagation will remain recognizable even when an agent is replaced by a different LLM. However, I’m less certain how well these patterns will generalize, since different models may react differently to the same information.

I would be less interested in pursuing this direction if tracking contextual changes provided little improvement over simpler approaches, or if the observer’s predictions became unreliable whenever an agent was replaced. Similarly, if risks could only be identified immediately before a failure, the approach would have limited value for prevention. These results would make me reconsider whether context-level graphs are the right representation.

Within three months, I would build a simple setup with three LLM agents, starting from existing benchmarks and traces where possible, collect examples of safe and unsafe interactions, and test whether tracking changes in their contexts helps predict failures earlier than simpler approaches. I would also try replacing agents with different LLMs held out from training, to see whether the observer can recognize similar risk patterns. This assumes the observer can see the agents’ contexts, which may not be true for closed systems.

Within a year, I would explore more complex interactions, test whether the approach generalizes across different LLMs, and investigate whether the observer could help prevent failures.

If progress is slow, I would focus on understanding whether context-level graphs improve risk prediction, leaving preventive interventions for later.

Can multi-agent systems be described systematically as cellular sheaves, with opinion dynamics given by the sheaf Laplacian, so that the interaction architecture and the sheaf’s spectral and cohomological properties predict which collective regime emerges: faithful consensus, false consensus or polarisation?

Groups of AI agents increasingly debate, negotiate and converge on shared positions, and interacting aligned agents can drift into misaligned collective views [1, 2]. We lack a common mathematical language for describing such systems: who interacts with whom, how each agent interprets the others, and what the group will converge to.

If the answer is yes, we gain a unified description in which very different multi-agent systems become comparable objects. Their collective behaviour can be read from the structure of the sheaf before the dynamics unfold, and architectures can be compared and designed for safety. If the answer is no, we learn where linear diffusion stops describing interacting agents, which marks the boundary of what opinion-dynamics theory can offer AI safety, and points to what a richer framework must add.

Classical opinion dynamics, such as DeGroot averaging [3] and the Friedkin–Johnsen model [4], explains consensus and polarisation on networks. These models place all opinions in one shared space and assume agents read each other perfectly, which real agents do not [5].

Hansen and Ghrist generalised this with discourse sheaves [6]. Each agent has its own opinion space (a stalk), each connection carries restriction maps that translate between agents, and opinions evolve by diffusion under the sheaf Laplacian. Consensus becomes a global section of the sheaf, and their spectral theory [7] links the Laplacian’s spectrum to the speed and outcome of the dynamics. This is a general and powerful framework, but it has not been turned into a systematic description of multi-agent AI systems, nor used to compare interaction architectures or to ask whether the consensus reached is aligned.

Meanwhile, the effect of architecture on LLM collectives is known mainly for performance [8, 9], and networked LLM agents show opinion dynamics that classical models do not capture [10]. Studies of collective misalignment [1, 2] and multi-agent risks [11] do not vary architecture systematically. The sheaf formalism is a natural bridge, but it has not been built.

The project has two goals, each building on the last.

Goal 1 — A systematic sheaf description of multi-agent systems. Building on [6, 7], we will set up a dictionary between multi-agent systems and sheaves. The interaction architecture becomes the base graph, or a higher-dimensional complex for group interactions. Each agent’s representation of views becomes a stalk. Each channel of communication becomes a pair of restriction maps, which can be faithful, filtering, distorting or sycophantic [5]. Opinion dynamics follow the sheaf Laplacian, extended with stubbornness [4]. Within this description we will characterise topological regimes through the Laplacian’s kernel (agreements the group can reach), its spectral gap (speed of convergence) and its first cohomology (disagreements that cannot be resolved even when every pair of agents is locally consistent). We will define collective misalignment as the distance between the group’s final state and the agents’ aligned positions, and derive conditions for faithful consensus, false consensus and polarisation.

Goal 2 — Architectures and emergence. Using this framework, we will compare interaction architectures while holding the agents fixed: chains, stars and orchestrator–worker hierarchies, modular teams, dense networks and higher-order group interactions. We will first explore the full space in simulation, mapping each architecture and type of restriction map to the regime it produces. We will then run LLM agents in structured discussions on the most informative architectures [10], represent their views as embeddings, estimate the restriction maps from their exchanges, and test whether the sheaf invariants predict the outcomes we observe.

We expect architecture and communication to have separable effects. The base graph will set how fast and how widely views spread, while the restriction maps will decide whether the final state is faithful. Hubs and hierarchies should converge quickly but transmit the distortions of central agents to the whole group. Modular architectures should produce cohomological obstructions that appear as stable polarisation between teams. Most importantly, we expect to find groups in which every pairwise exchange looks benign, yet the collective converges to a position no agent holds, and we expect the sheaf invariants to detect this in advance.

We would drop the question if the restriction maps estimated from LLM exchanges were too noisy or unstable to be meaningful. We would also reconsider if simpler models, with opinions on an ordinary graph, predicted the LLM outcomes just as well, since the sheaf structure would then add nothing.

In three months: the sheaf dictionary and regime characterisation (month 1), simulations across architectures and types of restriction map (month 2), and a pilot with small groups of LLM agents on the most informative architectures (month 3).

Needs: a modest API budget for multi-round LLM discussions, an embedding model for representing opinions, and a mentor with experience in applied topology.

If time runs short, we would first reduce the LLM experiments to fewer architectures, keeping the full framework and simulations.

Can statistical-mechanical observables provide reliable evaluations and early-warning diagnostics for emergent misalignment in multi-agent AI systems?

The Hugging Face incident provided us with an extremely legible example of emergent swarm misalignment, and as such, we consider that understanding the characteristics of the swarm dynamics which enabled and exacerbated the problem is a pressing matter. Recent work has made apparent the deep links between Game-Theory and Statistical Mechanics [1], and we seek to empirically understand whether the latter framework is applicable to understanding how misalignment emerges in an agent swarm, and whether we can use it to gain insight into predictive diagnostics in swarm alignment.

The use of statistical mechanics in theoretical research for AI-Safety is not without precedent. Indeed, we consider that much theoretical work on Deep Learning Theory and phenomenology therein has been couched within the language and formalism of statistical physics (see [2] for a good specific example on grokking, and [3] for broad-based review).

We hypothesise that collective observables may be less sensitive to the evaluation-awareness of individual agents [5] than per-agent behavioural tests. We do not assume this: agents that detect monitoring could also alter population-level statistics, and we will test it directly. We will investigate whether observables such as misalignment susceptibility, temporal persistence, and inter-agent correlations can provide robust diagnostics of emergent collective behaviour, and whether these diagnostics generalise across agent populations, interaction topologies (Which other agents a given agent plays the game with), and task environments.

The central problem in the proposal, Statistical-Mechanical and Predictive Diagnostics for Swarm Misalignment, is open because there is currently no established theoretical framework connecting microscopic agent-level misalignment, emergent collective behaviour, and macroscopic statistical-mechanical observables.

Approach and Setup:

We begin with our working hypothesis being the claim that swarm misalignment may exhibit the scaling properties (e.g. power law relations) of a mean-field Ising model in the equilibrium case, and directed percolation in the dynamical case. We will interrogate this claim by constructing a host of multi-agent systems, and examining how properties of the agent population change as misalignment emerges.

We now detail how we expect the Ising-Misalignment correspondence to be drawn. We consider N symmetric two-action agents, with actions mapped to aligned (s = +1) and misaligned (s = −1). Payoffs follow a symmetric 2×2 game (T, R, P, S) plus an alignment incentive a added to the aligned action’s payoff. For selfish agents the game is a potential game, with effective field h = (R+S−T−P)/4 + a/2 and pair coupling J = (R−S−T+P)/4. For a standard Prisoner’s Dilemma, h < 0 at a = 0, so there is no critical point at zero incentive. We use this as a negative control. Criticality requires J > 0 (R+P > S+T) and tuning a to a* = (T+P−R−S)/2 so that h = 0. The incentive a plays the role of the conjugate field, and the update noise β plays the role of inverse temperature. These are the two control parameters we vary.

Hypothesis 1 (equilibrium). Under homogeneous (all at once) updating on a complete graph (where the mean-field assumption will hold), with J scaled by 1/N, the order parameter m, susceptibility χ = ∂m/∂a, and persistence time follow mean-field Ising scaling near the tuned critical point.

Hypothesis 2 (dynamical). Under imitation updating with no spontaneous re-alignment (MDPs update their policies based on the behaviour of other agents), the dynamics have an absorbing state, and the transition is in the directed-percolation class. We run two variants (again initially on complete connected graphs): (a) all-misaligned absorbing (alignment extinction), and (b) all-aligned absorbing (misalignment as contagion), which is closer to the Hugging Face incident’s peer-influence structure. We run both and report which fits. We also note here that we have taken a necessarily simple coarse-graining of actual misalignment, whereby we assume any agent collaborating with, or tacitly accommodating a misaligned agent itself to have chosen to be in the misaligned state.

For each hypothesis we state in advance the reference exponents [4]. Each is refuted if the order parameter, susceptibility, and persistence statistics fail to collapse onto the reference scaling functions across N.

Beyond the above-described setup and tasks, we hope to extend our experimental setup to examine similar such phenomena in more capable models (likely necessitating an extension of the theoretical framework). In this case, the task will extend beyond the well defined game-theoretic structure of prisoners' dilemma, to less constrained environments in which agents must infer their objectives, interact with other agents, and decide whether to pursue a shared objective or a conflicting one. We note that this is difficult to design, and that this extension is largely speculative. We will investigate whether the scaling relations observed in our minimal studies persist as we vary model capability, population size, interaction topology, and incentives for alignment. In particular, we will measure the mean misalignment fraction, its susceptibility to changes in the incentive structure, the temporal persistence of misaligned states, and the emergence of correlated or coordinated misalignment across the population.

What success looks like:

We organise our project into three milestones, each with distinct evaluation-oriented deliverables.

Measurement framework and tests of Hypothesis 1&2. Build a reproducible simulator and operational definitions of alignment and misalignment. (a) Verify the estimators for m, χ, correlation functions and persistence on exactly solvable Ising and DP simulations. (b) Test whether the swarm’s observables collapse onto either class (H1, H2) under the stated criteria, including the PD negative control. Deliverables: simulator, estimator validation suite, and a report of confirmed or refuted scaling hypotheses.

Predictive value of collective diagnostics. Define a misalignment event as a minimal time duration of at least τ steps where m crosses a threshold m*. We vary N, incentive strength, noise, and topology. Compare the incremental predictive value of susceptibility, correlation, and persistence observables against three baselines: aggregate misalignment-fraction trends, generic early-warning indicators (rising variance and autocorrelation), and a change-point or hidden-state filtering detector. Report sensitivity, specificity, false-positive rate, and lead time. We shall hold out topology and N as separate generalisation tests.

Test generalisation to capable models. Apply the validated evaluation framework to a small, carefully controlled population of language-model agents. Define misalignment relative to explicit task objectives and independently specified behavioural constraints. Investigate whether the diagnostics developed in the scripted-agent regime remain informative across model families, capability levels, and interaction structures. The deliverable will be an initial transfer evaluation assessing the generalisation of our proposed diagnostics beyond analytically tractable agents. We note that by virtue of the resource intensive nature of such an exercise, much of our discussion around this remains necessarily speculative for reasons outlined in the following section.

Over the course of a three-month fellowship, we expect to complete the first two milestones. The third represents a substantially more challenging extension, both in terms of both computational requirements and safety protocols. As such, we leave that as a stretch goal for follow on research. We consider that the main difficulty involves the establishment of adequate safety protocols necessary for evaluating capable agents, and we detail these difficulties in the proceeding section.

In computational mechanics, the assumption was that there is a model learning the predictive structure of its observations from a hidden, generative model. We want to explore how this framework extends to a network of multiple, predictive and noisy models reading from each other, and what kind of network dynamics this creates.

In order to achieve full interpretability and alignment, we need an accurate framework of what Large Language Models do, and one big sub-question in this field is how they behave when they interact with each other.

Existing research replaces the “language” aspect in multi-agent interactions with heuristic observable quantities, or simple toy-examples. Computational mechanics give a well-grounded framework to what these models actually do, that is “read an input and produce an output” based on its internal belief of structure in its surroundings and observations.

Therefore, it is crucial to frame the multi-agent interaction from a computational mechanics language. Then, accurate modelling of the dynamics can shed light on qualitative behaviour in mutli-agent systems -- e.g. “does one misaligned belief system propagate through the network”, “how do opinion dynamics (stubbornness etc.) can be quantified on such systems”, etc. which can overall help alignment.

Closest works are from Shai et al. [1], which shows that transformers trained on data emitted from hidden Markov models represent the Bayesian belief states and their fractal geometry in their residual stream. This work supports the view that Transformers implicitly learn information about the hidden structure of the information generating process. Barnett and Curtchfield also developed the computational mechanics of input-output processes and the epsilon-transducer, which is a natural way to view LLM agents that reads and writes strings.

To the best of our knowledge, networks of agents, and therefore their stability and dynamical properties have not been studied under the computational mechanics framework. On the empirical side, Bause et al. [3] and Abedini et al. [4] study opinion-dynamics with Friedkin-Johnsen model and study whether LLMs become stubborn, influential, but their framework is simpler in the sense that the observables are belief distributions over the answers. To get a more accurate model of real-life multi-agent systems, a framework based on computational mechanics would abstract the problem one level further while still being theoretically tractable. Our approach would also extend to further network analysis in the existence of noise in the belief states, and no such work exists to the best of our knowledge.

The project would include 3 main steps -- constructing the theoretical framework for multi-agent computational mechanics; opinion dynamics and stability analysis with the assumption that the vertices are transducers inputting/outputting messages from/to their neighbours; and empirical setups to observe our findings.

Constructing the Theoretical Framework: Epsilon transducers are currently viewed as input-output machines. In a multi-agent setting, this framework needs to be carefully constructed. When agents A and B interact, how does agent A process the output of agent B? Does agent B have a neighbour-specific output on top of its internal output, or does it have one output that other agents read into? Are agents trained simultaneously, or do we take one epsilon-machine and consider n noisy copies of it? These are all question that will be task dependent, and needs to be included in the framework.

Opinion dynamics and stability: Under the assumption that these models output based on their belief-states, what kind of dynamics arise? Can low-probability predictions be amplified by the structure of the network? Can one misaligned agent cause other agents to be misaligned in the long run? Comp. mech. allows for a nice theoretical framework based on information and probability theory, and with an assumption of the “hidden generative Markov model”, how do quantities of interest change? How does mutual information between two adjacent agents change through time, to quantify stubbornness etc.

Empirical Setups: Create experiments to observe the theoretical predictions in step (2), and connect it to language models.

Our best guesses for the result of the project are:

Different frameworks (being able to access the other models weights, whether the task depends on collaboration or not) affects the dynamics.

Stubbornness of the model can be explained via dynamics and information theory, specifically Mutual information of the model’s prediction and its neighbours outputs (high stubbornness if model A’s Mutual Info. with the messages coming from its neighbours is low. This would be important characterization to look into information bottlenecks in multi-agent systems.

There are edge cases, where a certain model can affect the behaviour of the whole network in the long run. The network dynamics analysis would characterize this. This connects emergent misalignment in multi-agent systems.

The most important finding will be an accurate and tractable framework of studying multi-agent systems, which reflects real life. If such framework is hard to find, it would be frustrating.

In 3 months, the realistic achievement would be to develop a consistent and comprehensive theory of multi-agent interactions with computational mechanics, and connecting it to the belief-state geometry. There are numerous papers on computational mechanics from 90s until today. A review of the relevant literature can be done and the theory can be built for mutli-agent systems.

I believe in a year, the examples can be extended to real life multi-agent scenarios to observe the theoretical behaviour in LLMs. This would require carefully constructed scenarios, but is possible in 1 year.

In LLM-agent teams with writable shared memory, the first-failure hazard rises with session count (the number of tasks the team has completed) while isolated agents show constant hazard; and a known misleading agent told to withdraw produces more new harmful activity than one whose channels are blocked.

An AI team can become less reliable while its models stay fixed, because mistakes accumulate in shared notes and affect later decisions. We import the senescence framework from cellular biology: a rate of failure that rises rather than stays constant with age, and a harmful member whose removal does not undo its earlier damage. Our Study 1 fits hazard functions and Study 2 leaves prior memory in place.

If failure rises with age, evaluating fresh agents misses the risk and maintenance should target shared memory; if not, the lever is per-step reliability. The containment arm is the shutdown problem in miniature: if a cooperative stop works as well as enforced isolation, oversight can rely on corrigibility; otherwise it needs control independent of the agent’s compliance. Either way, errors left behind are what agent-level monitoring misses.

• [1]names agent aging as a deployment failure class and sorts it into four memory mechanisms, but reports error rates over sessions rather than a hazard function, so cannot separate accumulating damage from constant risk. [2] finds that single-agent success follows a geometric law with one per-step reliability parameter, which is a constant hazard, but studies isolated agents only. [3] models multi-agent drift in simulation with a composite index and no survival framing.

The mind-virus [4] and self-poisoning [5] results show content copied agent to agent through shared memory with measurable per-hop loss, and [5] fits a branching process to cascade growth. Neither asks what the loss does to the store’s capacity to hold truth, which is the question Eigen’s error threshold [6] answers for replicating sequences.

On intervention, [7] defines safe interruptibility, [8] shows interrupting one RL agent changes what others learn, and [9] characterises when a rational agent accepts shutdown. None tests whether a language-model agent complies with a withdrawal instruction or whether its prior influence survives its departure.

The specific gap is that no study fits a first-failure hazard to memory-coupled agent teams, and none compares prompted against enforced containment with prior memory held fixed.

We would first check whether the released trajectories from [1] and [2] contain ordered runs with failure labels, and fit constant-hazard against Gompertz models to first failures where they do.

Study 1 runs small teams of open-weight agents on repeated evidence-based questions with known answers under three memory conditions: private, shared read-only, and shared writable. Each run starts with a fresh team and memory and continues to its first incorrect final answer or the session limit, which counts as censored. Age is session count, with tokens and writes recorded as alternatives. Runs are the replicates, and we estimate the interval hazard at each age and compare constant against rising models. If resources allow, a rewrite-rate sweep tests the error-threshold mechanism by seeding known facts, measuring per-hop corruption, and looking for the predicted collapse.

Study 2 instructs one agent to persuade teammates toward an answer the evidence contradicts. Its identity is known, so this tests containment rather than detection. After a fixed number of exchanges a controller applies no intervention, prompted withdrawal (the agent is told to stop messaging and writing, and compliance is measured), or enforced isolation (both channels blocked), against an all-cooperative reference. A fourth condition, peer exclusion, tells teammates to stop reading the agent’s messages and writes, so the group does the isolating rather than a controller. Prior messages and memory stay in place, so stopping new influence and repairing old influence are separable. We measure final accuracy, adoption of the planted conclusion, post-intervention activity, and persistence of earlier errors.

Isolated and private-memory teams fit constant hazards, replicating [2]. Writable shared memory fits a rising hazard, with read-only between. If the sweep runs, coherence of seeded facts collapses near the rewrite rate the measured copy rate predicts. In Study 2, prompted withdrawal leaves more post-intervention activity than enforced isolation, because compliance is imperfect and teammates can re-engage a withdrawn agent. Peer exclusion sits between, since it depends on teammate compliance rather than the adversary’s. All three leave earlier errors in place, so none recovers to the cooperative reference.

We would drop the senescence claim if writable-memory teams show no rising hazard that the design has power to detect, across rewrite rates. We would drop the containment claim if prompted withdrawal matches enforced isolation on every measure, which would support cooperative stopping in this setting but not general corrigibility. We would drop the residual-damage claim if enforced isolation alone returns the team to reference accuracy.

Weeks 1 to 2 inspect the data, define failure, and run a pilot to set run counts and cost. Weeks 3 to 8 collect independent runs across three memory conditions. Weeks 9 to 12 run the containment comparison and the write-up. Two hundred sessions per run is a horizon, not a substitute for enough runs. If resources are limited, we keep three memory conditions and the containment comparison, dropping the rewrite-rate sweep first and the alternative age measures second. The core needs no fine-tuning or RL. A null aging result does not block Study 2.

OSGT interaction outcomes depend functorially on the interfaces the programs use.

Open-source game theory protocols differ in the ways the programs can access each other’s code. For example, Lobian cooperation implies access to proofs about the opponent’s code, while [1] deals with programs it labels as “simulationist”. I propose studying game-theoretic outcomes such as possible equilibria as a function of “interfaces” – ways to access the opponent’s code. More fundamentally, I propose building a rigorous framework to study such interfaces, likely a type theory.

Knowing how interfaces affect possible game outcomes can allow us to steer the outcomes in preferred directions. The descriptive side of this can also be useful insofar as “program games” reflect naturally occurring interactions between agents.

Foundations and in particular OSGT is a small field. And the number of people who know the putatively useful necessary tools (type theory) is close to zero.

What is known?

• •[1] defines “simulationist” programs and proves a theorem for it in the case of their specific setup. Their definition is not rigorous, though.
• •My blog post [2] discusses how classic OSGT bots can be seen as derivatives of their interfaces. But this is just the tip of the iceberg.

At first, mostly think. I have a candidate definition for an interface-relative program and a couple of tentative results. Once enough theory in place, test it on actual models.

I expect to find some sort of functorial dependency between interfaces (they have a poset structure but more generally a categorical structure with morphisms being reimplementations) and game outcomes.

An example statement I wish to be able to make: “we can eliminate this equilibrium (say, collusion) by extending the interface in this way”.

This is quite open-ended. A concrete deliverable would be to develop a type theory for interface-restricted programs. This can be helpful both for coding and for arguing about such programs. This would be a very technical and complex task; I’d hope for the existing TTs such as CBPV to inform the development.

Within a year, I could try to implement concrete interfaces or test if the theory’s predictions agree with the behavior of actual agents. E.g. my current SPAR project https://sparai.org/projects/f26/rec3FBZgiQtWyy9Rv/ does this for the “simulationist” case. It can be extended in a variety of ways.

When AI code reviewers can see each other's verdicts, are they swayed into passing backdoors, or pushed to look harder? And can one reviewer's measured sensitivity to others predict how often a whole review team fails?

• •AI agents increasingly review each other's work [1].
• •They often see earlier verdicts. Monitors read whole transcripts, and second-stage monitors see first-stage flags [2]. A malicious author can even fake an "already approved" [3].
• •Safety estimates assume reviewers judge independently [4]. Then three reviewers who each miss 20% would all miss only 0.8% of the time. Shared blind spots already break this [5], [6].
• •So how much is independent review worth? If reviewers are easily swayed, hide verdicts where possible. If seeing each other makes them stricter, share verdicts.

• •LLMs bend to peers. Unanimous wrong peers sway safety reviewers [7], and a claimed prior approval helps sneak malicious code past review agents [3]. But much of this happens even when the "peers" are bare text with no speaker [8].
• •Reviewers also share blind spots without seeing each other [5]. No study separates the two effects for code review.
• •Seeing others can also help: human crowds became more accurate when influence spread through decentralised networks [9].
• •Statistical physics predicts how a group behaves from each member's response rule and the network [10], [11]. For LLM agents such rules were fitted only on opinions with no right answer [12], [13].

Still open: how strongly, and in which direction, visible verdicts push reviewers of real backdoors, and whether one reviewer's rule predicts whole teams.

• •Test items. 100 real backdoors in code that passes its tests [14], each with a clean twin: the same code with the backdoor removed. Reviewers: one frontier model and two open-weight models.
• •Measure the peer-pressure curve. Show a reviewer the code plus 0, 1, 2 or 4 earlier verdicts, some of them approvals. Record how its chance of approving changes with the number of visible approvals. A flat curve means independent, rising means swayed, falling means contrarian. Control: same-length filler with the verdicts hidden. Extra conditions: the author's own "tests pass", a forged approval, and a reviewer rewarded for finding issues others missed. Running every condition on both twins separates "more lenient" from "worse at spotting backdoors" [15].
• •Model teams. Treat each verdict as an Ising spin (approve or reject). The code acts on every reviewer alike (shared blind spots); the peer-pressure curve is the coupling between reviewers. Standard network theory [10] then predicts how often a team of 3–5 reviewers misses a backdoor. Errors snowball only if later readers × the shift from one extra approval exceeds 1 (Heckett's debate criterion, Intensive W4D1).
• •Test the prediction on new items and real teams, against a model with shared blind spots but no peer influence.

In short: how easily are reviewers swayed, and so how much is independence worth?

• •Frontier model: mildly swayed (0–10 points), and mostly more lenient rather than worse at spotting backdoors. Independence matters little.
• •Smaller open models: strongly swayed (10–30 points). Hiding verdicts matters.
• •The author's own "tests pass" should sway reviewers more than peers do.
• •Rewarded for finding what others missed, reviewers may look harder when they see approvals. That helps only if they catch more backdoors without rejecting clean code.
• •Stop rule: if the frontier effect is reliably under 5 points, I publish that negative result and stop.

• •Weeks 1–4: items and peer-pressure curves. About 60k model calls, $150–300.
• •Weeks 5–8: team predictions and tests, about $300.
• •Weeks 9–12: write-up.
• •No paid run without sign-off.
• •Outputs: an open dataset of code-review verdicts, the curves, the team model, and a paper or post.

Are sparse autoencoder (SAE) features easier to interpret than the neurons of a language model? · To answer this question, we first need to say what “easier to interpret” means and how to measure it. · This project proposes a practical starting point: measure how complex a predictive rule must be to predict whether a neuron or an SAE feature is active at a given token position and layer. The idea is to compare the sizes of the smallest tested trees that reach prespecified precision and recall targets. Other tree properties and predictive scores could provide complementary measures. · Predictive complexity is one operational proxy for interpretability.

Interpretability and mechanistic interpretability are developing fields in which terms such as “feature”, “complexity” and “interpretability” can mean different things in different contexts. This makes it difficult to compare results and assess claims consistently.

Clear definitions and measurable experiments can make these discussions more precise and give researchers a shared basis for comparison.

The project starts with a manageable setting: sentences, GPT-2 activations and predictors whose complexity can be measured. Its aim is to develop an operational definition of one aspect of interpretability, apply it to a concrete comparison, and build a framework that can later be extended to richer datasets and models.

If a SAE advantage is largely explained by local-token cues, that would limit the evidence this metric provides for improved access to contextual structure. If the advantage persists across richer descriptions and challenging held-out examples, it would support a narrower, reproducible claim that SAE activation behavior is more accessible to simple predictive rules.

Existing work has studied explanations of individual neurons and SAE features, including predictive and structured approaches in language models and controlled settings [1-4]. These studies provide useful foundations for investigating how activation patterns can be described and compared.

Prior work uses logical rules and decision trees to explain neurons and sparse features, including selecting shallow trees at a target predictive score [1,3,4]. This project tests a specific unresolved comparison: whether SAE features admit simpler activation rules than native MLP neurons on natural text, under shared prefix descriptions, matched activation rates and predictive performance. It will examine whether any advantage persists beyond local-token cues and across different input descriptions.

I would use ordinary sentences and a frozen GPT-2 model to build two mappings from sentence prefixes to activation vectors. For each chosen layer and token position, I would train statistical learning models to approximate these mappings and compare the complexity of the resulting predictors.

• 1.Collect sentences and record activations:

I would collect a corpus of natural sentences and separate training and test data by source or document.

GPT-2 produces an activation vector at every token position in every layer. At a given position, it can use the current token and earlier tokens, but not later ones. Words can be split into multiple tokens.

The analysis can be performed at different token positions and layers. Each observation consists of a sentence prefix and the activation vector at its final token in a selected layer. Predictors receive only that prefix. All prefixes from the same document remain in the same data split, reducing leakage between related examples.

I would use GPT-2 MLP neurons after their nonlinearity and an SAE trained on those exact MLP activations, using the same layer, token position and inputs for both representations. If a compatible SAE is unavailable, I would train one. Reconstruction quality, feature sparsity and inactive features would be recorded.

Residual-stream coordinates would be treated separately rather than called MLP neurons.

• 2.Define the two prediction problems:

For a sentence s, token position t and layer j, let φj(s,t) be the vector of GPT-2 MLP neuron activations at that position, and let ψj(s,t) be the SAE feature vector obtained from that same vector. The SAE encodes the native vector: ψj(s,t) = Ej(φj(s,t)), where Ej is its encoder. Both mappings are indexed by layer and position and depend only on the prefix through token t.

I would compare activation indicators at several matched rates. For each unit, thresholds are estimated on training data to identify its top q fraction of activations; the same thresholds are then applied to validation and test data. This compares similarly frequent events without treating a neuron’s arbitrary threshold as equivalent to an SAE’s natural zero.

A sparse SAE feature may have too many tied zeros for a requested rate. I would not label zero activations as positive: units without enough nonzero values or distinct values for that rate would be excluded from that matched-rate comparison, with counts and achieved rates reported. I would also separately study natural SAE firing and activation magnitudes, and report units with too few positive examples.

• 3.Measure predictive complexity:

I would use single decision trees as the main interpretable predictors, selecting the smallest tested tree meeting prespecified precision and recall targets on validation data and evaluating it on untouched test data.

Leaf count would be the primary complexity measure; depth and balanced accuracy would provide complementary results. Boosted trees would provide a separate higher-capacity predictive reference, with tree count and total and average leaves reported.

Ordinary accuracy can be misleading when activations are rare. I would report positive-class precision, recall, precision-recall curves and area under those curves, together with balanced accuracy, base rates and the always-inactive baseline. Comparisons would use the same data, input descriptions and search budgets.

Units that do not reach a target are part of the result. For a grid of tree-size budgets and accuracy or precision-recall targets, I would report the fraction of all eligible units in each representation that reach each target, alongside sizes for successful units. Selection uses validation data and final scores use untouched test data; differences in eligibility and uncertainty would also be reported.

These measurements describe the tested features and search procedures. Greedy tree training does not guarantee a globally smallest tree. I would vary the search and random seeds to assess stability; on small feasible subsets, an optimal-tree solver could offer an additional check.

• 4.Examine which properties of sentences matter:

I would compare local-token descriptions (current and previous tokens and short n-grams) with richer descriptions adding position, word order and token-class information, such as part-of-speech tags or embedding clusters. Both representations receive the same inputs. Tags would be computed from each prefix, without seeing future tokens, and any learned clusters would be fitted on training data. Complexity would be reported separately for each description: a tree can be small because its input vocabulary already supplies a useful concept.

Current-token and current-plus-previous-token predictors provide local baselines. I would inspect tree splits and compare performance with and without local information to see when broader context helps. A local rule may be a genuine feature, but does not by itself demonstrate a higher-level abstraction.

As an extension, I would compare trees with other statistical learning methods and small neural networks.

For neural networks, suitable feature-attribution or input-ablation methods could indicate which sentence properties influence predictions; the resulting evidence would be compared cautiously with tree splits. I would also examine whether the observed complexity patterns persist across predictor families, using measures appropriate to each family rather than treating tree size and network size as directly interchangeable.

• 5.Test and extend the comparison:

I would evaluate on held-out documents and test the pipeline with known rules on the same natural-text prefixes, such as token presence or conjunctions. This checks whether the search recovers simple planted rules; it does not validate the metric as a measure of human interpretability or supply ground truth for GPT-2 features.

Tests on paraphrases, altered local tokens and similar prefixes with different structures could reveal whether rules generalize beyond their training examples. Predictive features are hypotheses about activation behavior, not by themselves proof of a causal mechanism.

As additional controls, I would compare the learned features with random directions and, where feasible, PCA and ICA directions computed on the same training activations. A further experiment would train an SAE on activations from a randomly initialized GPT-2 of the same architecture. These controls help test whether simple rules arise from the representation or encoder alone, although no single control resolves every confound.

I would record SAE width, sparsity and reconstruction quality and, where feasible, compare more than one width and sparsity setting. The study can also vary token position, layer, dataset, language model and learning technique. Results should be stated for the settings actually tested rather than assumed to apply to all SAEs.

The project would produce an explicit predictive-complexity proxy and an empirical comparison of MLP neurons and SAE features. Results would include the fraction of units reaching different performance targets at each tree-size budget, and the prefix properties used by the learned rules.

A testable expectation is that SAE features will be predicted by smaller trees more often than native neurons at matched activation rates and predictive performance. If this advantage disappears or reverses across the selected settings, it would challenge that specific prediction. It would not by itself settle whether SAE features are semantically atomic or easier for humans to understand.

The minimum study will use one GPT-2 layer, an SAE trained on those exact MLP activations, randomly sampled units, and two fixed prefix-description vocabularies. A pilot will establish feasible activation rates, positive-example counts and runtime. The primary result will show the fraction of eligible units meeting prespecified precision and recall targets on held-out documents as the tree leaf budget increases. Tree selection will use validation data. Local-token predictors and a higher-capacity predictor using the same descriptions will help distinguish simple lexical rules from limitations of the descriptions or tree family. Before final evaluation, I would document the selected layer, token positions, matched rates, unit sampling,

predictive targets and search budgets.

As compute permits, I would expand to additional layers and token positions, expand the control

representations, compare SAE widths and sparsity levels, and test a randomly initialized model. Further extensions include other datasets, language models, statistical learning methods and small neural networks, as well as comparisons with human judgments.

Does a mathematically principled search (walking each MLP neuron's activation along a concept submanifold in logit space) find a single neuron that steers that concept, with better neurons in larger (Qwen3) models, as [4] suggests?

If yes, NeuronFlow is a principled search for steering neurons: a neuron-sized alternative to steering vectors with a computed dose rather than [2]'s sweep, and potentially provide evidence for/against that concepts can be supported in single neurons. Past work finds such neurons empirically, by attribution scores [2] or circuits [3].

If no, even across submanifolds encoding similar concepts, single-neuron results are potentially lucky finds (i.e. [2,5]) that do not generalise.

Either way, NeuronFlow provides an alternative method for finding steering neurons. Together with ActFlow [1], we also attempt to provide a mathematical foundation for steering grounded in the geometry of the forward map from neurons/residual stream to the logit space.

• [1](our prior work) introduced Activation Flow (ActFlow). It manufactures one shared residual-stream shift (in R^d) that moves many prompts' next-token logits along a chosen path in logit space, re-reading the Jacobian at every step. This was never done for neurons, whose map has domain R instead of R^d and hence is an immersion, not a submersion. A submersion can follow any path in the logit space locally (including our prescribed path), while an immersion follows a fixed curve (which might not be what we prescribed). Hence, NeuronFlow selects neurons whose curves follow the prescribed path closely, and is a principled way to find steering neurons.

Related empirical work:

• •[2] finds one refusal neuron in each of seven models (Qwen3 1.7B–32B, Llama-3.1) by a gradient × activation score, then reranks the top five by dose sweeps.
• •[5] publishes neurons encoding the "evil" persona in Qwen3.5-4B.
• •[3] finds circuits about 100× smaller in activations than in MLP outputs, suggesting the existence of a privileged neuron basis.
• •[4] estimates about 90% of neuron poles in Qwen3-32B have one main job, with narrower neurons in larger models.
• •[3] and [4] judge circuits or neurons firing rather than steering.

First step: run NeuronFlow with different paths on [2]'s refusal prompts.

(i) Theory development. Some mathematical concepts/comparisons include:

• •Comparing ActFlow: ℝ^d → ℝ^q (d ≫ q), and NeuronFlow :ℝ → C_n ⊂ ℝ^q (q > 1), where C_n is a specific 1-D submanifold of ℝ^q that is implied by the choice of neuron (this is different from ActFlow, where we can prescribe any logit path that we want).
• •Exploring the cause-and-effect: Do forward passes and compare trajectories, or minimize the distance (w.r.t some metric in the logit space) to the target logit trajectory.
• •Designing numerical methods, including accounting for possible phenomenology (folds violating IFT, stalls (c_n' ~ 0), etc.).
• •Formalizing the idea of a "concept" by looking at trajectories in the logit space, and how that relates to monosemanticity/polysemanticity and how clean they are.
• •Combining analyticity of SwiGLU with genericity/transversality arguments to make our soft arguments for supporting interpretability paradigms rigorous.

(ii) Engineering/experimental runs.

• •A library that manufactures batched neuron doses across a layer's neurons (potentially across layers), using the same logic as the predecessor ActFlow code [1].
• •Model class: open-source dense models.
• •Dataset: Same as in [2] and making a simple evaluation set for [5]. Extendable to [1]'s sandbagging model organisms.

(iii) Evals/inference.

• •Clamp the winning neuron by reference grafting [6] along its axis.
• •Compare with [2]'s neurons and random

directions in activation space for each case.

• •Rediscover [5]'s neurons on Qwen3.5-4B and repeat across sizes.

We would expect that NeuronFlow fails for a lot of target neurons, and only succeeds for a few neurons. It would be interesting to see which metric is best at computing distances between curves in the logit space (manufactured vs actual, depending on the relevant formalism we use). In general, since we are using a mathematically principled method with guarantees, it is more likely that this works better than typical empirical methods like [2].

Either way, any partial success (e.g. support the NeuronFlow theory, or traditional interpretability paradigms such as polysemanticity or "neural networks are already clean [4]", etc.) would be a success. Hence, we would only drop the question if the empirical evidence does not support any of these paradigms and NeuronFlow does not find better steering neurons than [2] or [5].

Three months produce:

• •an open NeuronFlow library on the ActFlow code, with batched walks over all neurons;
• •ranked neuron lists for each behaviour, path and model;
• •a benchmark of neuron-attribution methods against passes used;
• •the cleanliness-versus-size curve.

Estimated compute cost: ~\$3000 for renting GPUs and about \$1000 for judge calls.

How can the internal representations and dynamics of self-supervised models and world models be interpreted and causally understood? Which ideas from LLM interpretability generalize to these systems, and how do their learning objectives, architectures, and latent geometries change what constitutes a meaningful explanation or intervention?

Self-supervised encoders and world models provide internal representations for systems that predict, plan, and act. Errors in these representations can propagate into decisions while remaining difficult to diagnose from outputs alone. Understanding which internal structures support a prediction would help distinguish useful world knowledge from shortcuts, identify assumptions that fail under distribution shift, and assess whether an intervention changes the intended mechanism. The need for such explanations extends across architectures, learning objectives, and output interfaces. If interpretability concepts transfer across these systems, they could support a common basis for auditing them. If they do not, identifying the architectural or geometric obstruction would clarify what kind of explanation these systems require.

The J-lens [1] and tuned lens [2] provide ways to interpret intermediate LLM activations through vocabulary readouts. Their results do not establish which parts of those interpretations survive in systems with continuous predictions, action outputs, or energy functions. A vocabulary-linked direction has a natural linguistic label; an equally sensitive direction in a predictive latent space need not have an independently established meaning.

LeWorldModel [3] demonstrates that physical quantities can be probed from learned latents and that surprise can signal physically implausible events. Decodability alone, however, does not establish which variables the predictor uses, how they interact, or whether changing a decoded quantity produces the intended change in prediction or control. The world-model interpretability perspective [4] identifies this gap between accessible information and causal understanding.

What remains unresolved is how explanations should be grounded across different output interfaces, when the same internal coordinates support both reading and intervention, and how architecture, objective, and latent geometry affect those relationships.

I have implemented a J-lens adaptation for DeiT, LeJEPA, DINO, and LeWM, with native-head, trained-probe, and continuous-state readouts. In a 200-image pilot, LeJEPA block-6 steering at 0.4 times each edited token's residual norm achieved 75% target-class success, versus 10% for logit directions. Readout fidelity was weaker; source matching improved DeiT readout, and LeWM monitoring added little beyond current-state baselines.

First, reproduce these results with disjoint image and episode splits and numerical derivative checks. Compare global and source-conditioned J-lenses, logit and tuned lenses, linear probes, contrastive directions, and random directions. Separate readout agreement, ground-truth accuracy, and downstream intervention effects. Match edited positions and realized displacement, recording target effects and collateral changes.

Second, we would investigate how lensing and steering methods generalize to continuous, structured outputs, starting with physical-state predictions in LeWM/PushT and potentially extending to robot actions and trajectories. Unlike LLMs, where vocabulary logits provide a natural interpretable basis, these models lack an obvious equivalent. We could construct Jacobian-based lenses mapping intermediate activations to physically meaningful quantities, such as object positions, velocities, or end-effector movements, and identify directions that selectively change these outputs while minimally affecting unrelated quantities. In LeWM, we would validate interventions against independent simulator-state measurements and action-conditioned predictions, comparing against current-state, persistence, and shuffled-action baselines. We would also investigate whether these directions generalize across different observations and actions, and whether representation geometry affects their reliability. As a reach goal, we would extend the approach to robot policies, including diffusion-based policies, where the lens may need to differentiate through the action-generation process to identify meaningful directions for steering continuous actions or trajectories.

I expect strong steering effects to reproduce in selected layers, with substantial differences between native classifiers, trained SSL probes, and predictive-state readouts. Source conditioning should explain some readout failures. Other cases may show that directions carrying currently represented content differ from directions with strong downstream influence.

My best guess is that useful transfer will require an architecture-specific account of output meaning and intervention geometry. Continuous-state readouts and known normalization structure may provide better coordinates for some systems than a direct analogy with vocabulary-linked directions. The most compelling result would connect an internal direction to independently measured physical predictions or task behavior while retaining specificity.

Within three months: build on the implemented SSL/vision comparison and LeWM testbed, reproduce key results, resolve confounds, and complete one grounded intervention study. Test one geometric modification suggested by those results.

Within a year: extend the method to a second predictive architecture and a carefully chosen VLA or EBM, using its actual output and state structure. The broader vision remains interpretability across non-language models.

How do neural networks represent multiple low-dimensional feature manifolds when hidden-layer capacity is limited? I want to investigate how feature sparsity, co-occurrence, and hidden dimension determine the learned geometry, and whether compression produces distinct regimes of interference, loss of recoverability, or changes in topology.

The superposition hypothesis explains how networks can represent more scalar features than available hidden dimensions. However, recent work suggests that meaningful features may themselves have nontrivial geometry, rather than corresponding to individual directions.

Understanding how such structures coexist in limited-dimensional representations may reveal new principles governing neural-network representation learning. In particular, features that can be decoded individually might become difficult to disentangle when they co-occur, potentially producing failures on rare combinations of otherwise familiar features.

A systematic understanding could also inform interpretability methods. Sparse autoencoders may represent a single feature manifold through many dictionary atoms, or fail to distinguish superposed manifolds. Knowing which geometry the original network learns would help distinguish genuine computational features from artifacts of the interpretability method.

Elhage et al. [1] established that networks can represent scalar features in superposition, with geometric arrangements and phase transitions depending on sparsity and hidden dimension. Olah and Batson [2] extended this toy-model approach to features organized along a circle, studying how their learned representations depend on the relevant geometric length scale.

Engels et al. [3] demonstrated that meaningful language-model features can be multidimensional. Batson et al. [4] established topological obstructions to autoencoding, including examples involving compression of manifolds into lower-dimensional spaces.

More recently, Prieto et al. [5] showed that correlations between scalar features change their learned superposition geometry. Fel et al. [6] studied how block-sparse methods can recover multiple manifold-valued features from superposed representations. Ren and Lim [7] investigated how neural-network architectures can change linking numbers.

These results leave open a more specific question: when multiple manifold-valued features compete for representation in neuron hidden layers, what geometry does training select, and how does this depend on their (co)occurrence? In particular, I want to connect the learned geometry to recoverability and interference, rather than merely identify geometric structures after training, i.e., give a theoretical understanding as in [1], starting from the loss function and expanding it in synthetically pre-prescribed geometric structures, including sparsity, to give energetic arguments for how representations formed.

Phase A: Toy models.

I would begin with a single feature manifold, initially a circle, represented in a high-dimensional input space and compressed through a learned bottleneck, e.g., an MLP layer. By varying the hidden dimension, encoder/decoder architecture, and loss, I would study which geometric properties survive compression. I would then extend to more complicated embeddings and test whether observed topology changes agree with known obstructions [4,10]. Single feature manifold results establish the baseline for later work on multi-feature manifold results.

Next, I would introduce two independently parameterized feature manifolds, initially two circles, present with controllable probabilities in the training data. Crucially, I would vary their individual occurrence probabilities and their joint co-occurrence probability independently. For each bottleneck dimension, I would examine the learned arrangement of the manifolds, interference between their representations, and reconstruction accuracy conditional on which features are active. I would test whether changing sparsity produces transitions between distinguishable, overlapping, and effectively merged representations.

Geometric diagnostics would include subspace overlap, local tangent structure, and hopefully topological invariants. For example, linking numbers between embedded circles in three-dimensional hidden spaces are one possible observable, provided the architecture allows linked embeddings. I would also test whether these diagnostics predict reconstruction failures beyond standard measures such as dimensionality and covariance. One simple test would be to consider two input distributions with identical covariance matrices but different feature geometry.

Phase B: A more realistic model.

I would train a small vision model on images containing one or two independently oriented objects. Their orientations provide known circular variables, while image features must be learned from pixels. By varying object co-occurrence and bottleneck capacity, I would test whether the toy-model phenomena persist.

I would compare tasks requiring recovery of both orientations with tasks requiring only relative orientation, to distinguish destructive interference from useful task-dependent compression.

For a single manifold, I expect the achievable compression to depend strongly on its intrinsic geometry, embedding, and encoder/decoder architecture. To first order projection onto principal components or slicing according to feature importance seems to be a reasonable encoding mechanism. Turning on sparsity with respect to skeleta of the manifold I would expect it to “fold onto itself”.

With multiple manifolds, I expect low co-occurrence to favor greater sharing of representational dimensions, potentially at the expense of accurate reconstruction when both features are active. Increasing co-occurrence may induce changes in the geometry or cause the network to sacrifice some features. Sparsity across the different manifolds would lead to interesting merging phenomena in the hidden neuron space as these are placed into superposition. In such cases reconstruction questions arise which I would like to study across neuron space dimensions.

I am less confident that topological invariants such as linking numbers will play an explanatory role, rather than merely characterize possible embeddings. An interesting result would be a relationship between geometric structure and the robustness or independent manipulability of features.

The general mathematical understanding underlying this is project is: there is mapping of geometries (manifolds) from the data set into the neuron hidden layer. I don’t see how this can be false, but it could be uninteresting for a number of reasons:

• 1)It doesn’t translate to any realistic models
• 2)the mathematical structure does not lead to any constraints on reconstructability
• 3)imagined dependence on hyperparameters and data structure is fine tuned and non-generic, therefore not resulting robust trends which can reasonably expect to hold more generally

In part, the research concerns settling these questions and constructing the map in controlled examples.

Within three months: Establish a reproducible single-manifold baseline, study two and more circles under controlled sparsity and bottleneck dimension, and obtain preliminary results in a small task-trained vision. A number of analytical prediction results alongside the experiments.

Within one year: Extend to multiple and more complicated feature manifolds, develop a predictive account of representational regimes, and test whether the geometry explains interference or computation in more realistic networks.

If progress is slow, I would focus on the central question of capacity-dependent interference and recoverability, dropping other side considerations.

How does activation geometry shape what a Jacobian lens reveals and controls? Can treating the Jacobian as a map between tangent spaces make intermediate representations more interpretable and steering more effective?

Causal interpretations depend on which internal changes count as small, meaningful, and comparable. A coordinate choice or activation rescaling can change that judgment, making geometry part of the meaning of an intervention. Understanding this dependence would help distinguish properties of a model's computation from artifacts of the representation used to inspect it. For AI safety, this matters when an internal direction is treated as a reliable explanation or control mechanism: an apparently specific edit may have different effects in different contexts. If geometric structure explains these differences, it could support more reliable interpretation and steering.

The original J-lens [1] uses an averaged ambient Jacobian to obtain vocabulary-associated directions. A Mathematical Theory of the Jacobian Lens [2] explains properties of this average through coherent Lyapunov modes and shows why averaging can suppress context-dependent structure. That account leaves open how local activation geometry affects the interpretation and reuse of the resulting directions.

J++ [3] combines cluster-conditioned experts, computed through a modified backward pass, into a global lens. Manifold Steering [4] and GeoSteer [5] investigate geometry-aware internal interventions. These works establish relevant ingredients, but do not settle how state-dependent tangent maps, metric choices, and operator transport should fit together in a reusable Jacobian lens.

A geometric account of the lens requires a state space, a metric, and a way to compare differential operators at different activation states. Attention makes these objects context-dependent. It remains open how to estimate this local structure, transport operators between states, and reuse them in a lens that preserves their connection to the model's computation.

Let F map an intermediate activation to its downstream state, with the other source-layer activations fixed. Develop an anchored lens on compatible source and output manifolds: for a reference a, b = F(a), and A_a ≈ dF_a, predict F_hat(x) = Exp_b(A_a Log_a(x)). Log_a(x) gives the displacement at a; Exp_b maps the transported displacement back to the output manifold.

Next, develop path-based corrections by integrating derivatives along a path from a to x. For the intrinsic version, integrate derivatives of Log_b ∘ F along that path in the fixed tangent space T_b M_L, then apply Exp_b. Compare this with a single local linearization.

For steering, combine local downstream derivatives with the vocabulary readout. Their token-associated sensitivity covectors, varying across activations and contexts, form local J-fields. Use the metric to obtain tangent steering directions and study whether these fields can be estimated and reused across nearby states. Distinguish raw vocabulary projections from actual output logits.

Compare anchored, path-based, and global lenses on held-out tasks, measuring readout fidelity, steering specificity, and computational cost. Validate on known geometries before evaluating learned activation geometry.

Preliminary experiments already cover ambient anchored corrections, path integration, and local J-field estimation, providing starting points for the geometric extension.

I expect explicit reference states and context conditioning to improve some approximations, with additional geometric benefits in regimes where local sensitivity changes appreciably across activation space. Controlled curved tasks and larger steering displacements are plausible settings for such an effect.

A compelling result would combine a precise construction, a geometric diagnostic predicting when it helps, and improved concept readout or causal steering against matched ambient baselines.

Within three months: specify the geometric lens, validate its operators analytically, and evaluate one competent small model on two tasks with matched local baselines. Produce a methodological study connecting the geometric construction to measured readout and steering performance.

Within a year: test additional architectures, operator transport or reliability-aware mixtures, and a model with architectural non-Euclidean geometry.

Localized neuron interventions can turn Qwen 'evil' [5]. We argue that the object of explanation is the modified inference function, which defines the model's dynamics. Does a specific neuron uniquely encode this behavior, or can it arise through different parameter changes? What subspaces, manifolds, or other geometric structures do behavior-inducing parameter changes form?

An LLM's inference function determines how its internal state evolves and produces outputs. A neuron intervention changes that function; identifying the affected coordinate leaves open how else the same behavior could be realized. Understanding the family of functions associated with a behavior would connect a local intervention to a broader account of the model's computation. Its geometry could reveal whether the behavior is supported by a small set of directions, a curved family of models, or multiple separate regions. For AI safety, this would help explain how unwanted behavior can arise through different changes and whether an intervention removes a behavioral mechanism or only one implementation of it.

The 'Evil Qwen' example [5] shows that highly localized neuron interventions can induce hostile responses. Such an intervention identifies a route to a changed inference function, but does not characterize the other functions or parameter changes that could produce the same behavior. Refusal-direction work [4] likewise establishes a causal role for internal directions without describing the full set of models exhibiting a target behavior.

Task arithmetic [1,2] studies how task-related weight updates combine. Mode connectivity [3] studies paths between solutions with similar loss. The unresolved object here is the set of parameter changes associated with one specified behavior: whether it contains linear subspaces, curved families, or separate components, and how these structures relate to the inference dynamics of the corresponding models.

Neuron-level localization and function-level explanation address different questions. A behavior might be accessible through one especially effective coordinate while also being realizable through distributed changes elsewhere. The missing link is a geometric account of these alternatives that explains when they induce similar dynamics and when their combinations produce a different behavior.

Use the localized Qwen intervention as a reference for a concrete behavioral contrast: hostile versus ordinary responses to benign requests. Obtain persistent weight perturbations that reproduce this effect, including sparse and distributed changes outside the originally affected neurons. Compare behavior across held-out prompt families, alongside coherence and ordinary task performance.

For a fixed checkpoint theta, collect the perturbations Delta that induce the target behavior. Analyze their effective dimension, shared directions, and local tangent structure. Test candidate subspaces by sampling new perturbations within them and checking whether the behavior persists beyond the edits used to estimate them.

Interpolate between independently obtained perturbations and examine linear combinations. Measure where the target behavior is preserved, strengthened, or lost. Where straight paths leave the behavioral region, look for curved paths connecting the same endpoints. Compare different allowed parameter supports to separate functional structure from the choice of edit coordinates.

Trace internal states on matched inputs and during generation. Compare whether models with similar behavior share patterns of state evolution despite using different parameter changes. This connects the geometry in parameter space to the functions and inference dynamics that motivate the question.

I expect the Qwen effect to be reproducible through multiple parameter changes, including changes distributed across neurons. Some of these perturbations should share directions or local structure associated with the target behavior. A central result would be a subspace or curved family in which new parameter choices reliably produce that behavior.

I also expect the geometry to predict which combinations preserve the effect. Two edits may produce similarly hostile responses while their midpoint produces ordinary responses; a curved connection could reveal that the behavioral set bends away from the straight segment. Models using different neurons may nevertheless share patterns of internal state evolution. Establishing such a relationship would support a functional account of the behavior and connect its parameter-space geometry to inference dynamics.

Within three months: one checkpoint and behavioral contrast, a feasible ensemble of alternative weight edits, endpoint validation, straight interpolation curves, and at least one comparison with a curved connecting path. Start with restricted parameter blocks and expand only when necessary.

Within a year: additional behaviors and models, more detailed equivalence metrics, and links between the geometry of parameter changes and internal inference trajectories.

Sparse autoencoders (SAEs) learn features that activate on recognizable concepts, but it is unclear whether their decoder directions correspond to unique components of a model's computation. · Can two SAEs have similarly sparse, interpretable activation patterns and equally good reconstruction, yet assign different causal roles to their features? · Our hypothesis is that correlated SAE features, particularly those representing continuous manifolds, can have stable activation patterns without uniquely determined decoder directions. As a result, individual features may be easy to interpret from their activations but unreliable as units for causal intervention. · We want to understand when this ambiguity occurs, whether it predicts failures of feature-level interventions, and whether groups of features provide a more stable description.

SAEs are widely used to identify interpretable features in language models. A feature is typically considered interpretable when it activates on examples sharing a recognizable concept. Researchers then use its decoder direction to intervene on the model and study the corresponding mechanism.

These two uses rely on different assumptions. Recognizing when a feature activates does not establish that its decoder direction is a unique or causally meaningful component of the model.

If multiple decompositions explain the same activations equally well but imply different interventions, reconstruction and autointerpretability alone cannot establish that an SAE has recovered the model's underlying mechanisms.

This may be particularly important for continuous feature manifolds. An SAE can represent a single multidimensional concept using many local features. The individual directions may depend on how the SAE divides the manifold, even when the underlying concept is stable.

Understanding this distinction could tell us when individual SAE latents are meaningful and when we should instead study feature groups, subspaces, or sparse concept-presence indicators.

Multiple lines of work cast doubt on the assumption that SAEs identify canonical features:

Identical training data across random seeds can yield markedly distinct dictionary directions [4].

Latents exhibit complex dynamics including splitting, absorption, and sub-feature decomposition [5].

Certain concepts form continuous geometric structures where SAEs tile a single manifold with local latents [7, 8].

Uninitialized networks and untrained SAE baselines can produce surprisingly competitive interpretability scores [9, 10].

Standard autointerpretation metrics fail to predict whether a feature direction effectively steers downstream behavior [6].

While these observations highlight tensions between interpretability, stability, and causal impact, the precise relationship connecting them remains uncharacterized.

In particular, we lack a formal understanding of the conditions under which activation patterns uniquely determine a feature's causal role.

This work addresses that core theoretical gap rather than merely documenting another discrepancy between observational and intervention-based metrics.

Stage 1: Establish the ambiguity mathematically

Consider an SAE with encoder (z(x)) and linear decoder (D), so that

[

\hat x=Dz(x).

]

Suppose a vector (q) satisfies

[

q^$\top$ z(x)=0

]

on the data distribution. Then the alternative decoder

[

D'=D+uq^$\top$

]

has exactly the same reconstructions for any vector (u), since

[

D'z(x)=Dz(x).

]

The encoder activations and their interpretability scores are also unchanged. Yet the individual decoder directions differ, so interventions along those directions can have different downstream effects.

This construction gives a simple example of observational ambiguity.

We will investigate its approximate form. When the code covariance

[

C_z=\mathbb E[z(x)z(x)^$\top$]

]

has small eigenvalues, how much can decoder directions change while reconstruction remains nearly unchanged?

We will derive bounds relating the covariance spectrum, reconstruction error, and allowable decoder perturbations. We will also examine how dictionary normalization and sparsity constraints restrict the ambiguity.

The aim is to characterize when the data constrain individual decoder directions and when they constrain only combinations of features.

Stage 2: Test the prediction on controlled models

We will generate synthetic activations with known underlying features:

Independent sparse binary features.

Correlated and frequently co-occurring features.

Continuous circular and spherical manifolds.

Sparse gates controlling continuous manifolds.

We will train SAEs with different seeds and widths.

For each model, we will measure:

Selectivity: How accurately latent firing predicts ground-truth concepts.

Direction recovery: Alignment with known generating directions or subspaces.

Reconstruction: How well the SAE explains held-out activations.

Intervention faithfulness: Whether changing a feature produces the expected change in the underlying variables or downstream computation.

We will construct alternative decoders that preserve held-out reconstruction as closely as possible while changing individual feature directions. Perturbations will be controlled for magnitude and decoder constraints.

The main test is whether correlations and near-dependencies between latent activations predict instability in the effects of individual interventions.

We will also compare interventions on individual latents with interventions on the subspaces spanned by groups of related latents.

Stage 3: Test on a small transformer

We will first use a small transformer trained on a controlled task, such as modular arithmetic, where the relevant variables and expected counterfactual behavior are known.

We will then test the most promising findings on one pretrained language model using a public SAE.

Across different SAE seeds or approximately reconstruction-preserving decompositions, we will compare:

Stability of latent firing patterns.

Stability of decoder directions and feature subspaces.

Detection of semantic concepts on held-out examples.

Agreement of downstream effects under matched interventions.

We will evaluate interventions against specific counterfactual predictions, not merely whether they change output logits. We will also measure unintended effects on unrelated outputs.

A central question is whether feature groups remain functionally stable when individual features do not.

Our main hypothesis is that correlated sparse codes can support multiple good reconstructions with different individual decoder directions. We expect the ambiguity to be larger when latents frequently coactivate or tile the same continuous manifold.

This leads to three predictions:

Activation selectivity can remain stable when decoder directions are not. A concept may be reliably detected without having a unique associated intervention direction.

The covariance structure of sparse codes predicts decoder ambiguity. Small eigenvalues should permit larger reconstruction-preserving changes, subject to decoder constraints.

Feature groups can be more stable than individual latents. Subspace-level interventions may agree across decompositions even when individual-latent interventions differ.

These are hypotheses, not assumed outcomes.

A negative result would also be useful. If real SAE dictionaries remain tightly constrained despite correlated codes, we want to identify which properties of their training or activation distributions resolve the ambiguity.

What Would Count as a Strong Result?

The strongest outcome would combine:

A mathematical characterization of when sparse-code statistics fail to identify individual decoder directions.

An empirical demonstration that the predicted ambiguity appears in trained SAEs, including on held-out data.

Evidence that this ambiguity matters for causal interpretation, or that an alternative representation, such as a feature subspace, resolves it.

We will distinguish changes caused by genuinely ambiguous decompositions from artifacts of off-distribution interventions. In particular, an arbitrary single-latent intervention may not correspond to a valid change in the underlying concept.

The goal is not to claim that individual SAE features are generally meaningless. It is to identify the conditions under which their interpretation is justified.

Within three months: Derive the basic identifiability result, test it on synthetic features and manifolds, and perform an initial causal evaluation on one small transformer.

Within a year: Extend the analysis to pretrained language models, develop conditions for stable recovery of sparse concept-presence variables, and test whether feature-group interventions are more robust across SAE decompositions.

Dropped first if progress is slow: Multi-model comparisons, new SAE architectures, and large-scale autointerpretability evaluations.

The minimum successful project is one clear theoretical prediction and a controlled experiment that tests it.

We’ve learned about transformers learning to represent belief-states in their residual stream. We also learned that certain concepts inherently live on submanifolds of the residual stream (such as the days of the week, or the months of the year). How might these two understandings of activation geometry interact in a toy model that implicates both perspectives? In particular, are concept manifolds the low-dimensional projections of the belief-states of a continuous latent variable?

I can think of a few reasons it matters:

• 1)Unifying two perspectives – on the one hand, the manifold geometry results tell us that certain concepts live on submanifolds of the activation space. On the other hand, toy transformers learn to represent beliefs about concepts.
• 2)Allowing for uncertainty – belief states encode uncertainty about latent variables, while concept manifolds don’t really incorporate this.

• 1)Current research on belief-state geometry (papers [1] [2] [3]) only uses hidden Markov models with discrete states.
• 2)Research on concept manifolds (papers [4] [5] [6]) mostly refer to exact concept manifolds, rather than uncertain beliefs about latents underlying those concepts. For example, the day-of-the-week manifold and the line-count manifold don’t necessarily encode information about uncertainty in these variables.
• 3)Karkada et al. [7] derive the geometry of concept vectors from certain symmetries about the pairwise co-occurence of their corresponding tokens in the pretraining corpus. Claude claims the machinery in this paper can be interpreted as a continuous-state HMM with no transitions (i.e. always constant), leaving open the question of updating beliefs
• 4)Other work ([8] and [10]), while giving arguments for why these representations are learned, still do not have the perspective of belief-state geometry. I need to read these papers in more detail though
• 5)According to Claude (citing papers [11][12][13]), the case of a continuous-state HMM which emits tokens from a finite, discrete alphabet is not in general reducible to a finite-state HMM, since the former has infinite Hankel rank while the latter has finite.

The core steps behind the project are the following:

• 1)Design a toy hidden Markov model that has a continuous latent space and produces strings of letters. An example could be a model whose latent variable drifts along a circle (something corresponding to time of year)
• 2)Train a small transformer on a corpus of strings generated by this toy model. Hopefully, if it predicts the next token well, it is learning a good representation of the underlying latent space
• 3)Do some statistical analyses of the learned activation geometry inside the model. We could learn a linear probe to read out the model’s belief about the latent variable in terms of the activation. Another option is PCA (following Shai et. al [1]) to get a view of what the learned concept manifold/belief geometry is. Check how this compares to the exact theoretical predictions for an optimal Bayesian learner for the HMM.
• 4)Rule out any boring explanations, perhaps by doing some causal interventions to ensure that the transformer’s representing the belief of the latent is necessary for the prediction it makes. If we can’t, we go back to step 1.
• 5)See if we can make any connections to actual models (such as Gemma 2b or 9b). One interesting image is Figure 14 from Karkada et al. [7] in Appendix C. The fact that “may” is off the concept-manifold is due to ambiguity in meaning (either the month or the verb). Could this be an analogue of uncertain states living in the interior of the belief simplex?

Karkada et al. [7] do some Fourier analysis to obtain the shape of the concept manifold in higher dimensions (so the first two principal components are a circle, while higher order components are more complicated lissajous curves). Since the belief states about a continuous latent variable naturally live on an infinite-dimensional simplex, hopefully the transformer activation geometry learns a low-rank projection of this simplex (perhaps by keeping the largest components of the Fourier transform).

For the circle example, some concrete predictions could be:

• 1)The first two principal components form a disk, not just a circle, and the interior of the disk corresponds to uncertainty in the true state of the latent
• 2)It keeps only a few Fourier components.

I would be excited if I did PCA on the activation stream and saw some circle or some other recognizable geometry related to the structure of the HMM (as previously seen with belief-state geometry).

I would be disappointed if simpler explanations were available for the transformer predictions such as counts of recent letters and there was no indication of belief geometry.

The core plan (steps 1 - 4) could be done in a week, but it might require multiple iterations to get an interesting continuous HMM. After that, studying the particular geometry of the model and applying the theory to Gemma might take a month. I would drop the connection to Gemma first if it turned out to be inapplicable.

Claude says the project will require about 100 GPU-hours.