Proposals

For two-labelled Markov chains on a path, a polynomial-time algorithm can decide whether changing each state’s transition probabilities within a given error budget permits an exact bisimulation quotient with at most a specified number of states.

[= For rational two-labelled birth–death chains M, deciding whether (minimum possible size of a ε-feasible partition) $\mathrm{OPT}_\varepsilon(M) \le k$ under unrestricted rowwise ℓ₁ perturbations is solvable in polynomial time in the input bit length, with rational ε and integer k given as inputs.]

A Markov chain describes random movement between states. Here, states lie along a path, transitions between neighbours are possible in both directions, and staying put is allowed. Each state carries one of two labels, both of which occur.

We may change transition probabilities, including adding transitions, provided the sum of absolute changes from each state stays within the budget. We then merge states sharing a label and identical transition probabilities to every merged group, the resulting model is a bisimulation quotient.

An efficient algorithm would find the smallest permitted abstraction. A hardness result would show why path structure alone is insufficient, motivating further restrictions or approximate optimisation.

Kiefer and Tang [1, Theorem 1] prove that deciding whether a general labelled Markov chain admits an approximate quotient of prescribed size is NP-complete, even for size five. Their reduction uses a state with arbitrarily many successors, so it does not establish hardness for birth-death chains.

Spork et al. [2, Section 4] study perturbation-based probabilistic bisimulation and its relationship to other approximate equivalences. Their model allows perturbations to introduce new transitions, a convention retained here.

The remaining question is whether restricting the original transition graph to a connected path makes minimum-partition selection tractable.

First, I would examine the known test for whether a proposed grouping can be made valid within the error budget and determine how the path restriction simplifies its constraints. Inputs would use rational probabilities and a rational budget. Here, polynomial time means that running time is bounded by a fixed power of the number of bits needed to write down the input.

I would pursue two proof directions. For an efficient algorithm, I would look for a property that every instance has an optimal grouping satisfying and use it to reduce the choices the algorithm must consider. The argument must allow distant states to merge.

For hardness, I would try to encode a known difficult decision problem into these chains. The construction must preserve two labels and transitions in both directions along the original path. I would prove that the constructed instance admits a sufficiently small quotient exactly when the original problem has a solution, accounting for every permitted change to transition probabilities.

Exact calculations on small examples would help test proposed structural properties and constructions before attempting general proofs.

My tentative prediction is that path structure permits a polynomial-time algorithm, although I do not yet have a theorem supporting this. I expect a successful algorithm to handle mergers between distant states explicitly.

A hardness proof would make me abandon the tractability prediction under the standard assumption that P differs from NP: some problems with efficiently checkable solutions cannot be solved efficiently.

I would drop the question as a new project if existing work already settled it under the same rules for changing probabilities.

After one month, I aim to have a checked account of the closest literature, reproducible code that finds exact answers for small examples, and either a proven property of optimal groupings or counterexamples explaining why a proposed algorithm fails. A complete classification is the larger goal.

Is there a polynomial-time algorithm that repairs a probability table by deleting at most a fixed constant times the minimum probability mass needed for perfect condensation?

Perfect condensation means that two observations share a label that can be determined from either observation, and become independent once that label is known. The label explains all their dependence.

Some distributions almost have this structure: removing a few unlikely outcomes may be enough to recover it. Others may require substantial changes. We want an efficient way to distinguish these cases.

Our algorithm would remove selected outcomes and rescale the remaining probabilities to sum to one. Its guarantee would bound how much probability it discards compared with the best possible repair. This would tell us how much of the original distribution supports an exact shared explanation.

Eisenstat develops the theory of condensation [1]. For two finite observations, existing results describe what perfect condensation looks like in their probability table: after rearranging rows and columns, the positive entries form separate rectangular blocks, with independence inside each block [2].

When all possible outcomes are equally likely, our problem becomes a known graph problem called Bicluster Deletion. An existing algorithm removes at most four times as many outcomes as necessary [3]. Unequal probabilities make things harder. We must account for both the probability removed and whether the probabilities remaining inside each block satisfy independence.

First, examine small probability tables with two or three rows and columns. Find the best repair by checking every possible set of outcomes to remove. These examples would help identify which patterns make repair easy or difficult.

Next, investigate whether small incompatible patterns can guide which outcomes to remove. The aim is to relate the cost of our choices to costs that any successful repair must pay.

One possible tool is linear programming: replace some yes-or-no choices with fractional choices, solve this easier problem, and try to turn its answer into a valid repair with a controlled increase in cost (also known as LP rounding). We would also look for examples showing where this approach fails.

I tentatively expect that the structure of probability tables will allow a constant-factor approximation: the amount discarded would be at most some fixed multiple of the minimum necessary.

I would change direction if examples showed that our proposed method consistently underestimates the true repair cost by a factor that grows with the table size. That would rule out this particular approach, but would not prove that every efficient algorithm must fail.

A useful negative result would explain precisely which feature prevents our method from working

After one month, I aim to have verified small examples and a proof supporting the approximation method or a counterexample identifying its limitations. A complete approximation algorithm is the longer-term goal.

Given a set of model outputs, what is its pre-image inside an LLM’s activation space? Does the pre-image form a simple, interpretable geometric structure? What are its dimension, curvature and topology?

When the pre-image is simple this could give a way to connect outputs to internal structure and may be useful for steering. If it is complicated/disconnected/unstable it would identify concepts that do not correspond to simple activation-space objects.

Prior work has found simple geometric structures for some concepts in LLM activations[1]. Other work has developed methods for tracing output directions or regions back through neural networks[2]. These approaches do not directly study the pre-image of a chosen output set. We propose to study this pre-image.

Choose a small LLM and a simple concept with known structure (e.g. days of the week). The goal is to construct a manifold in activation space that is the pre-image of outputs with support on only weekday logits, i.e. “the answer is one of the seven days”. We allow outputs that say (20% monday and 80% friday). Ideally we’d want to construct the pre-image across all/most layers.

As a first step we construct the pre-image of a discrete set of outputs: those with all the mass on one day of the week. This should reconstruct the known approximately circular representation from Engels et al. [1].

We then try obtain the pre-image manifold for outputs that are a mixture of different weekdays. If we can construct this manifold we would then look at its geometric and topological features: how curved is it? Is it simply connected? does the same shape persist across layers?

A useful reference could be [3] which Claude tells me approximates the pre-image of an output set, but used it for didn’t study any concept’s manifold or apply it to simple concept sets.

Success would be if we find concepts with a manifold that

• 1.give pretty plots
• 2.persist across layers
• 3.are low dimensional
• 4.are very ugly and/or too hard to construct

After a month it'd be nice to have constructed the pre-image for days of the week and verified that it contains the known approximately circular representation from Engels et al. [1]

After the end of the month it would be nice to study the geometric structure of that manifold.

For an asymmetric n-state Mess process where a transformer's residual width is less than the belief-simplex dimension ($d_{\text{model}} < n-1$ for n hidden states), how does it discard, or merge belief states; · 1.Similar whole-future prediction pairs? · 2.Close pairs geometrically? ·

Identifying if compression biases in training follow the information-theoretically optimal choice is necessary for making statements about what large models discard, after we understand the situation sufficiently for toy models and synthetic processes.

Analysing two compression mechanisms and the meaning of if a probe fails to recover a distinction, because either the model never represented it, or it represented it and then discarded it.

Future-optimal merges the pair of beliefs whose whole-future predictions are most similar. If this information was discarded then the distinction was predictively redundant and it’s informative.

Geometric merges the pair closest in the simplex, regardless of what they predict. The distinction may have mattered but was merged for the wrong reason, and the failure is misleading.

• [1]showed that transformers trained on hidden Markov processes linearly represent optimal Bayesian belief states, but always at $d_{\text{model}} = 64$, far above the 2–4 dimensions of the belief simplex, and named compression beyond the residual stream as crucial future work. [2] explained how attention builds these beliefs, again at $d_{\text{model}} = 64$. The work closest to answering our question is [3]: for processes built from parts, transformers use factored orthogonal subspaces, and when factoring is lossy they still collapse to it first and add dimensions only slowly, even when narrower than the joint belief space. But [3] studies decomposable processes, where one lossy form is singled out in advance. For a single non-decomposable process no candidate is given, so which beliefs get merged is open. Theory supplies a target: the information bottleneck for time series [4] gives, for each number of states k, the k-state compression of beliefs that discards the least information about the future, and [5] bounds how many belief features near-optimal prediction needs. Both measure cost in bits or states rather than linear dimensions, and neither has been compared with trained networks. [6] already named this memory–accuracy frontier as open.

First, for asymmetric Mess-n (n = 3–6), compute exact belief sets and two candidate compressions: future-optimal (minimising information loss about the future) and geometric/PCA (minimising distortion in the belief simplex). Keep processes where these predict different merges. Set context length using a filter-stability bound, and compute the entropy rate as the benchmark loss of a perfect Bayesian predictor.

Second, reproduce [1] and [2] on Mess3 using a small transformer with a beginning-of-sequence token, affine belief probes, and the papers’ controls. This validates the experimental pipeline.

Third, sweep residual width from 1 to n + 2 across several seeds. Measure prediction loss relative to the entropy rate, probe error at each layer, and the PCA spectrum of activations.

Fourth, below width n − 1, use the affine probe to identify which beliefs the model merges. Compare these empirical merges with the future-optimal and geometric predictions, and determine how the information-bottleneck state budget k maps onto network width.

Finally, train a small RNN with hidden size k as a control, where the memory constraint is explicit rather than architectural. If transformer compression matches the future-optimal prediction, test sharper predictions about which beliefs merge and in what order. If instead it resembles the architectural bias found in [3], characterise that bias and investigate why training produces it.

The key test is whether narrow transformers discard beliefs according to predictive similarity or geometric proximity, distinguishing information-theoretically useful compression from architecture-driven compression.

The different situations are structured as observation, interpretation and response.

1.At width ≥ n − 1, prediction loss is close to the theoretical best. The model needs at least n − 1 dimensions to predict well; narrower than that, something is forced out. The experiment functions.

2.When the model is too narrow to hold all beliefs, the pairs it merges are the ones whose long-run predictions are most similar, not the ones that are closest to each other geometrically. Model discards what is least useful for predicting the future. Main positive result.

3.At width 1 for Mess 3, it would lose some information and could analyse what is kept.

• 4.The dropped beliefs match neither prediction. The pattern looks like the compression bias in [3]: the model merges in a way that reflects the architecture, not the data.Redirect: characterise and explain the architectural bias rather than test the original prediction.

he study is restricted to asymmetric Mess processes with n = 3–6 hidden states, focusing on cases where future-optimal and geometric compression predict different merges. The first stage prioritises Mess3 and Mess4, with larger n used where computationally practical. The project focuses on identifying and comparing compression patterns, rather than developing a general theory of transformer representations. The RNN control and investigation of why a particular compression emerges are secondary and can be dropped if time becomes limiting. Symmetric or near-symmetric processes, larger state spaces, and broader architectural comparisons are outside the initial scope.

How does belief state geometry in the residual stream activations sharpen (or fails to sharpen) as we scale model size? By "sharpen" we mean "become closer to the theoretically predicted belief state geometry".

Computational mechanics makes falsifiable predictions about what internal representations of the models should look like, assuming optimal prediction. Targeting an instrumental goal as an intervention target point is scale-free: it will be relevant to both current and future models. In fact, we would expect larger, more capable models to be closer to optimal predictors, hence exhibit even sharper belief state geometry.

One objection to existing comp-mech research is that it is constrained to toy setups. Investigating how the sharpness of the belief state geometry scales with model size would help us understand how the how earlier findings from toy setups would carry over to real-world AI systems.

• [1]showed that training toy transformers on sequences emitted from HMMs gives rise to the corresponding belief state geometry emerging in the residual stream activations.
• [2]show that the same belief state geometry emerges in LLMs when the HMM sequences are provided in context. This is a step towards closing the gap between toy models usually analyzed in comp-mech research and real-world AI systems. However, [2] only analyzes in-context learning, as opposed to explicit training. No prior work conducts a systematic analysis of how the sharpness of the belief state geometry scales with model size. This systematic analysis could help us discover clear scaling laws that we might extrapolate to even larger models.

We will study how the sharpness of the belief state geometry scales with model size and with HMM complexity.

We measure the sharpness of the belief state geometry by computing $R^2$ of the affine map fit between the internal residual stream activations and the theoretical belief state geometry.

We will also try to come up with other metrics to quantify sharpness.

Setup A: trained-from-scratch toy transformers.

This setup allows us to precisely control the model size, as well as the size of the dataset used for training (which is difficult with real LLMs, where models of different sizes were likely also trained with different number of tokens/different compute budget).

Setup B: pretrained LLMs, HMM sequences in-context.

This setup is more realistic in that it focuses on LLMs.

We reproduce the setup from [1] on toy transformers of different sizes. We start by keeping the width/depth ratio fixed, and scaling the number of params. We train each model configuration on the same dataset of sequences samples from the HMM. We plot the $R^2$ of the linear fit against number of parameters (model size).

For the LLM in-context setup we do the same, measuring how $R^2$ scales along models of different sizes and the same model family.

Once we obtain the plots, we inspect them visually and try to fit some function to them -- a scaling law.

One hypothesis is that the sharpness of belief state geometry increases monotonically with model size.

Another hypothesis is that sharpness of belief state geometry first increases, and then decreases (sharpness is an inverted U shape function of model size) -- maybe once the model is expressive enough, it doesn't need to compress the information and it can encode the belief in a less compressed way.

A result that would make me drop the question is if there is no clean relationship to be observed between the sharpness (measured in terms on $R^2$) and the model size.

I think the whole project is doable within 3 months. If we don't have enough compute, we could cut the toy-transformer setup and only focus on the LLM in context setup, because it is less computationally expensive (we only do inference, no training).

Ambitious plan would include trying to find a theoretical explanation for the observed scaling relationships.

Suppose we are given a model which has been trained on a hidden HMM. By making suitable assumptions, is it possible to infer properties of the underlying HMM by randomly sampling the network and looking at the model’s internal activations?

In practice, the data generating processes models are trained on (e.g. language) is largely unknown, and any techniques which could shed more light here would be of interest. For example, more knowledge here could potentially have applications for safety as we will have a better understanding of how the model will generalise in any given context. If successful, this project would exhibit a toy example whereby we infer properties of the underlying data-generating process (e.g. state-transition probabilities) by sampling the network and using heuristic assumptions.

In [1] the authors train a transformer model on a known HMM and show that the model linearly represents the corresponding belief state in its activation space. This project can be approximately viewed as the inverse problem: knowing this is how transformer models behave when trained on HMMs, can we infer any distributional properties of the underlying HMM when it is kept hidden from us. To be clear, multiple HMMs give rise to the same dynamics, so our objective would be to characterise an appropriate family.

We still need to iron out the details, but we believe this is possible in principle. As a rough idea...

We can start with a simple setup: train a transformer (or a similar network) with data generated by a known stochastic process (e.g. a given HMM). We then assume that all we have access to is the transformer residual stream and the activations at each layer. First, we would need to understand the alphabet: this can be achieved by sampling the transformer output. Then, we can feed sequences of alphabet elements and, for each token, record the activations after the token is processed, the output probabilities and the actual token emitted. We can use these ingredients to reconstruct the state-transition probabilities. Assuming that the belief state is linearly embedded in the activation space, a PCA-style argument will give us proxies for the belief state, and then we can all of the information gathered thus far to realise a range of HMMs which have the same dynamics as the underlying data generating process.

Based on the belief-state geometry work of Simplex, our prediction is that it is possible to exhibit an example of a HMM whose dynamics are the same as the hidden HMM simply by randomly sampling the network and extrapolating internal beliefs from the models activations.

One of the main challenges could be recovering precise belief states from the activation vectors. Activations may encode more information than just belief states in a non-trivial manner. There is also some uncertainty around coverage: applicability of this study may be limited in cases where testing enough sequences is unfeasible due to the large number of possible states and tokens.

After the first month, we will have solidified the mathematical argument and conducted some initial experiments with very simple stochastic processes. We believe that if everything went smoothly this project would not take too long to complete.

I want to understand whether a small transformer trained on next-token prediction on hierarchical data linearly encodes in its residual stream the Bayesian posterior over the hidden nodes.

Deep networks and transformers beat the curse of dimensionality because real-world data is hierarchical and modular, and depth lets a network exploit that [4]. Belief-state geometry [1] predicts how a transformer represents Bayesian beliefs, but only for data without hierarchy.

If the prediction holds, the representation of hierarchical data becomes derivable from the data model, ie we can predict which level sits in which subspace at which layer and check against this ground truth. In the future, it would be exciting to extend these results to training to understand how transformers builds these levels of abstraction during training and with an incomplete hierarchy.

If the prediction fails, it will be informative to study how it fails. The network might only keep low-level evidence or a factored approximation that drops hierarchical correlations [6].

The existing Simplex line of work initiated in [1] showed that transformers trained on next-token prediction for hidden Markov models linearly represent the Bayesian belief over the hidden state. [6] showed that when a process splits into independent parts, the belief is stored in orthogonal subspaces. None of these works study processes whose hidden states are hierarchical.

Ref. [2] explicitly studies hierarchy. The authors train transformers on a fixed-tree model. They find that outputs match (exact) belief-propagation marginals, and the attention maps mirror the tree. However, their task is masked prediction, and intermediate layers are probed nonlinearly using a neural network for hard ancestor labels, not belief vectors (probing nonlinearly is not convincing). [3] finds that autoregressive transformers on grammars store parsing-chart states, again via nonlinear probes on discrete targets. [4] introduces a widely known model, the Random Hierarchy Model, which explains why depth helps learn hierarchical data through sample complexity, but this work does not examine model representations.

To my knowledge, no work derives the belief state for next-token prediction on hierarchical data, or tests whether it is linearly represented.

Using Bayes theorem I derive the belief state of the hierarchical model in [2] as the posterior over path nodes given the prefix. I then enumerate all possible beliefs to build the geometry and plot this geometry for simple, low-dimensional models.

Next I train a small transformer akin to [1] on next-token prediction and check that it learns the data to near zero loss.

At each layer, I linearly regress the residual stream onto different targets: the joint belief, the per-level marginals, and the pairwise marginals of neighbouring levels. The plan is to compare these different targets to show which representation the transformer learns at which layer.

Using the filtering parameter of [2] which removes correlations above a chosen level I check if the representation changes at those levels.

Eventually, I would like to extend the theory to more general models such as the Random Hierarchy Model [4].

My prediction is that some residual stream layer linearly encodes the Bayesian posterior with low regression error. This regression error is high initially and training reduces it progressively. Beliefs about lower tree levels become decodable at earlier layers, as the attention maps in [2] suggest.

What would make me drop the question: if the transformers reach near zero loss, but no linear map recovers the theoretically anticipated optimal belief state better than a random baseline.

In the first month: I would aim to derive the exact beliefs, train a transformer to near-zero loss, and use layer-wise linear probes to reconstruct the belief state for the three parameterisations.

What I need: a mentor to check my theoretical setup and derivations, the public code of [2], a GPU to train the transformer.

If turns out to be three years: I would first cut down on the generalisation ambition, and next the empirical work, and if that still turns out to be too big of a question, I would simplify the theoretical setting further.

Could we use perturbation theory to quantify the deviation in belief states of transformers when it learns data from indecomposable processes?

It enhances our understanding of internal representations of concepts in transformers.

It would provide a more robust link between the well-established tools from physics and the language used to study deep learning models from a theoretical perspective.

It lays the groundwork for other mech-interp studies.

Background: This project builds on [1], where it was shown that transformers have an inductive bias towards learning factored representations, even when this hurts prediction. In [1], conditionally independent generalised hidden Markov models (which I will call factored processes) were used for data generation to check if transformers learn factored representations. They also trained the transformer on a “perturbed” process that I call an ε-close factor process (or indecomposable process) to check if transformers prefer learning factored representations even when this hurts prediction. This project aims to compute the belief states in the representations learned by the transformer trained on data from the indecomposable process for small ε using perturbation theory. It also aims check if there exists a phase transition between factored and non-factored representations (when the transformer does transition to the non-factored representation regime for large enough ε).

I don't know of any papers that attempt to use perturbation theory to compute belief states and then compare with measurements from residual stream activations.

Ref. [2] is the seminal work on this specific framework on transformers and belief state geometries. Perturbed Markov processes have been studied in the context of bounding errors for numerical situations (such as [3]).

First week or two would involve a deep-dive of the relevant mathematical theory. This would predominantly entail studying the various appendices of [1] in detail. Towards the tail end of this process, I hope to start performing the calculations using time-independent degenerate perturbation theory to compute the perturbed belief state vectors that I expect to measure in residual stream activations when data is generated from these indecomposable processes. I also expect to start (collaboratively with other experts) constructing a dictionary between the language of computational mechanics and that of physics. The next few weeks would involve designing and running the experiments that extracts these belief states from specific models (which need to be at least large enough for the model to learn the non-factored or joint representation). The exact methodology would probably resemble that of [1], with perhaps some minor adaptations relevant to these measurements.

If my hypothesis is true, then for ε-close factored processes, the belief states that are measured should match the perturbed belief states that are calculated. Depending on the nature of the calculations, a phase transition at some $\varepsilon_{\text{crit}}$ is observed (or not). In any case, the notion of the representation attractor, “where small ε keeps predictive vectors near the product-state manifold while large ε pushes them off” [1], can be proven or disproven by this project.

I expect the theoretical predictions to be done in the first month. Making the physics <-> GHMMs + deep-learning language dictionary may take longer if some wrinkles need to be ironed out. I am not sure how long it would take to design and run the experiments, but I do not think there is any circumstance in which this project takes more than three years -- it was intended to be an incremental curiosity based on some works I found very fascinating!

Does the learning of belief state geometries exhibit phase transitions and if so, what are their order parameters and hierarchy across increasingly complex generating processes? ·

We aim to develop a more robust, quantitative understanding of the learning behaviour of belief state geometries regarding the order in which different types of structures are learned and internally modelled. This would lead to intuition for how LLMs learn throughout training but could also make predictions for in-context learning, for example the order in which different aspects (factors) of the in-context environment are learned, such as assistant conversation vs. testing environment, inferring personal details about their user, etc..

In [1], the authors showed that models prioritise factored learning, even when the ground truth was an interacting system which did not genuinely factorise. They showed that how fast the model adopted a higher-dimensional depended on the strength of the interaction term modifying the factorised transition probabilities (Fig. 4 in loc. cit.).

In [2], the authors showed that for non-ergodic processes (where the generating process is chosen randomly among a fixed set of generators), in-context learning of the model can be visualised as a probability distribution over belief states across the generating processes. Their data on the learning process displays evidence of a phase transition in the KL-divergence (Fig. 7 in loc. cit.)

We aim to combine and extend these findings, characterising in what order the two transitions happen given a non-ergodic, interacting generating process and how this depends on the strength of the "interaction".

The first goal would include reproducing the behaviour of the plateau in [2] and measuring its length against the interaction strength epsilon. This should give rise to a scaling law that could be matched against theory, as may be developed by our second proposal, or by an LLC estimator.

As a second step, we would investigate whether this adaptation happens in steps or as a single phase transition. Decomposing the interaction term into subspaces, we would expect subspaces with stronger interactions to be learned first.

A similar LLC analysis could also be performed on the learning dynamics of the non-ergodic process [1], although here a potential order parameter for the potential phase transition is much less obvious.

Lastly, we would combine non-ergodic and factorisation to observe (qualitatively) the order in which the transitions happen. We set up a generating process with K sources, each composed of $N_i$ factors. We introduce an interaction term and gradually increase its strength.

My current expectation is to see a gradual, step-by-step transition from the factorised to the interactive regime.

Intuitively, I would expect the non-ergodicity of the generating process to be fully learned before the phase transition to non-factorised belief states occurs (with low confidence).

The complexity of the transformer architecture with Adam training in [2] makes exact quantitative predictions difficult. We would cut the LLC estimations if they appear too difficult.

As outlined under "what you would do", the first step was to recreate the findings of [2] for various interaction strengths.

If this turns out to be much harder, I believe the subsequent steps can be omitted as necessary, in particular the combination of the factorisation and nonergodic processes.

How do LLMs represent uncertainty between discrete alternatives? We will test whether activation geometry is best explained by probabilities, log-odds, or Fisher–Rao geometry, and whether this generalises across concepts

A reliable understanding of geometry could improve belief decoding and steering. Understanding when it emerges could also clarify how models learn to represent uncertainty.

Previous work identifies geometric belief representations in synthetic transformers [1], curved belief manifolds in pretrained LLMs [2], and connections between geometry, uncertainty, and steering [3]. We could maybe build on this by testing how well uncertainty representations transfer between settings. Maybe see if there are commonalities

Start with a small LLM and simple tasks involving uncertainty between two alternatives, such as object identities or weekdays. Vary the uncertainty using stated probabilities and supporting evidence, then measure output probabilities and activations.

Clear visualisations and evidence that uncertainty representations transfer across contexts—or a clear account of where transfer fails. Extend to three alternatives or a second model if time permits.

By the end of the first month, we aim to have a working experiment on one small pretrained LLM, comparing probability, log-odds, and square-root probability descriptions of uncertainty between two alternatives. Deliverables would be layer-by-layer plots and quantitative comparisons on held-out prompts, including checks that the model tracks the intended probabilities.

We would need GPU access for inference and activation extraction, standard interpretability tools, and mentor feedback on the experimental design. If the scope grows, we would cut SLT, steering, and additional models first, keeping one categorical task and a reliable comparison of the three descriptions.

With RRXOR, we have 1-layer transformers get close to optimal loss but known one-layer mechanisms don't work. Does a 2-layer model split the belief update between its layers so that ablating its layer-1 previous-token heads would raise loss above optimal?

-Models infer latent variables such as user identity or persona from context. This covers which mechanism gradient descent picks when there are several options, and what happens when inference is split across layers. If the 2-layer model splits the update, the belief is assembled across several layers which means a probe at one layer could miss it.

-This also extends [2] beyond processes where every token is informative. If the second layer turns out to be useless this would indicate transformers prefer doing everything in one step instead of splitting the update into different stages when there’s enough width.

-Either way, the exact belief is known, so we can check whether ablations and probes can recover the real mechanism.

• [1]showed that transformers trained on HMM data store the belief states as a geometric object in the residual stream. For Mess3 it shows up in single layers and for RRXOR it only shows up when all layers are joined together. [2] covers how one layer does this on Mess3 i.e attention adds up one correction per past token, weighted by distance, and the MLP bends the result into the belief shape. [3] found on other HMM tasks that the MLP does the update and attention moves information. [4] showed that transformers combine neighbouring tokens across layers using previous-token heads. In RRXOR each token is 0 or 1 with probability 0.5 at every point in the cycle, so only groups of tokens can signify anything about the hidden state. The Mess3 mechanism hence doesn’t work, but early runs show a 1-layer model still gets close to optimal loss. None of these say what each layer contributes on RRXOR. This might be a general fact about parallel computation: [5] shows self-attention can’t model periodic finite-state languages unless the number of layers or heads grows with the input length, and RRXOR is exactly this kind of periodic process.

First: train small transformers with 1 and 2 layers on RRXOR using the existing code from [1], and compare their loss with the exact optimal loss. Setup similar to [1] i.e $d_{\text{model}}$ 64, Adam, context length 10 and 20, 3 seeds with different random initialisations.

Then look at the attention patterns and QK / OV circuits directly, so the hypotheses come from what the models actually do. Starting list:

• •One layer could record the whole sequence and let the MLP do the work
• •One layer could use attention that depends on which token is where
• •Two layers could split the work, with the first combining neighbouring tokens and the second gathering them

Find out which one each model uses by removing or simplifying parts of it and seeing where the loss goes up:

• •Token-dependent attention: replace each attention pattern with its average over token identity
• •Recording the whole sequence: swap the value path for one where token and position are just added together
• •Two-layer split: look for previous-token heads in layer 1 (most attention on $t-1$ / $t-2$), mean-ablate them and see which positions get worse

For each mechanism, measure how much of the loss gap it recovers. If none of them recover most of it, the real mechanism is outside our list.

Then vary depth, width (heads, $d_{\text{model}}$, MLP size) and context length to see when a second layer is needed. Confirm with activation patching, and check our setup on Mess3.

Prediction: a 1-layer model gets close to optimal loss by recording the whole sequence and letting the MLP do the work. A 2-layer model splits the work, with the first layer combining neighbouring tokens and the second gathering them, so ablating its layer-1 previous-token heads raises the loss a lot. What would change my mind: if ablating layer 2 in the trained 2-layer model barely changes the loss, the second layer is useless, which would indicate transformers prefer doing everything in one step instead of splitting the update into different stages. If none of the mechanisms explain most of the loss gap, I’m missing something on my list and would go back to looking at what the attention heads actually do.

After the first month: the 1 vs 2 layer runs on RRXOR with 3 seeds, the ablations for each mechanism, and the check on Mess3. The models are tiny (~50k parameters) so they train in minutes on a laptop, no GPU needed. If it turned out to be three years of work I’d cut activation patching and the width sweep first, and keep the 1 vs 2 layer comparison on RRXOR.

The question is two-fold: · 1) does an LLM represent a state of the world for a given prompt? · 2) if there is uncertainty about the actual state, is the distribution over possible states represented and how? ·

We will understand how LLMs represent world under uncertainty. This will allow for:

• •better monitoring (by tracking latent belief states related to (mis)aligned behavior) and
• •intervention (by understanding the geometry of belief).

Tracking beliefs will possibly give better mechanisms for:

• •tracking misaligned behavior (such as lying: if the model's output is inconsistent with its beliefs) and
• •understand model's biases (see if model assumes certain details by default, so more inclined to do certain things).

• [1]and general Simplex line only considers synthetic datasets and transformers trained on this data. It is not clear if the models do probabilistic inference on real data. [2] shows that models do represent latent state in form of propositions that bind entities together, but it doesn't cover the probabilistic state or more general ground states.

• 1.Reproduce [2] and iterate on: find how a deterministic state (i.e. simple propositional facts) is represented in the latent space.
• 2.Add partial observability/uncertainty. Here's one way to do the experiment: in the prompts, describe a setting where an object is hidden in one of given locations. Find how this is represented by probing: check what is the vector associated to a general prompt, and which are associated to prompt+answer. The hypothesis is that the latent vector of the input with uncertainty is a weighted sum of vectors corresponding to prompts with specified position of the object, which would mean that the uncertain state is a distribution over certain states.

3.1. If this is the case, try intervention: would steering change the output as if it was changing belief? Then test the mechanism in various settings and find if it generalizes, i.e. there are universal directions associated to uncertainty or alternative beliefs. Also try to find the belief forming mechanism, that is by the layers when the belief is "formed", what is the circuit that did that.

3.2 If this is not the case, try to find an alternative explanation. Perhaps genuine uncertainty is different from a mixture of beliefs, so the experiments would need to be split in two and the mechanisms described independently.

I expect that in certain, but not all, settings the state is indeed a distribution over several ground states. If there is no, even simple, setting in which a state can be seen, then it would indicate that uncertainty is represented in a different way.

I also expect to see that LLMs are not calibrated in their beliefs and are heavily affected by the training data (be it biases or simply noise), so the "default" distributions over ground states will be far from uniform.

I believe that all of the plan that I stated before is executable in one month. The main problem is to find an alternative mechanism if the suggested one is not supported by evidence.

Pretrained LLMs that predict in-context HMM sequences and linearly represent belief states can verbalise their next-token prediction, but their accuracy on questions about hidden structure (phases, length, content) stays at shuffled-control level across sizes.

• •When models can't share the structure they operate on, any oversight that depends on asking the model misses that aspect and underestimates the factors that influence behavior. We should examine those structural levels using internal probes rather than questions. This also establishes a baseline for genuine limitations: if a model fails to disclose something it utilizes, even without any motive to conceal it, that failure in other areas shouldn't be seen as evidence of deceit.
• •If models can report their structures, or begin to at a larger scale, asking them becomes a cost-effective, validated alternative to probes for uncovering hidden structure. The point at which they start to provide reliable self-reports indicates when we can trust them.

The two ingredients are presented and studied separately but never combined. [1] shows pretrained LLMs predict HMM in context in a practically optimal way. The belief state is shown to be linearly extractable from the residual stream and used for prediction. But they never ask the models. [2] shows in-context prediction again without asking. [3] asks the models and finds gaps between classifying and stating, but their rules are deterministic text classifiers, without any true internal state to check the report against. [4], [5] show that models can verbalise structures acquired in fine-tuning, not in context for pretrained stage. [6] shows that models propose rules that fail to apply. No work found puts prediction, representation and verbal report jointly in the same model and process, while what the model tracks is exactly known.

I’d kick things off by diving deeper into today’s pilot project. My focus would be on three main areas: first, I’d assess how accurately the model predicts outcomes compared to the ideal belief-state predictor; second, I’d check if it can clearly explain its next prediction when asked; and third, I’d see if it can handle straightforward questions about the hidden phases. To make sure the data is solid, I’d compare these results against order-1 and shuffled controls, using a larger number of seeds for more reliable insights.

Next, I’d run the same tests on the six models from [1], which range from 2B to 9B. Since their probes already shed light on what each model represents, I’d put together a table that outlines what each model predicts, what it signifies, and what it can articulate. This would help pinpoint where reporting starts to stumble as the models get bigger.

If I notice any breakdowns in reporting, I’d make sure it’s not just a matter of poor prompting by experimenting with different phrasings, allowing for more time to think, and providing a few worked examples.

Finally, I’d follow the belief directions from [1] and see if the model’s responses align with those changes.

I expect to see some dependencies of the ratio WellPredicted/WellExplained on the complexity of the HMM and the size of the LLMs. I would like to find some scaling laws for these accuracies on answers.

I’d let go of the question if the gap closes when the model is prompted correctly—like with better phrasing, more thoughtful input, or by providing a few examples of other processes. That would suggest it’s more about how we’re asking rather than a lack of underlying knowledge. I’d also reconsider if bigger models just do a good job of outlining the structure, because then there wouldn’t be a clear threshold to identify.

By the end of the first month, I'd like the core table done: the six models from [1] on the four representative HMMs, 10 seeds each, showing how well each predicts, whether it can say its next prediction, and how it does on the structure questions against controls, with a check for poor prompting. I'll need about one A100 for a few days (or Colab Pro) and the public code from [1]. Trying to check the affine map for the hidden states would be great, too.

I don't see how this could turn out to be three years of work.

Does the dimension $\hat{d}$ of a persona in a model change as model size grows? I expect it to saturate at the data's persona rank d, not at model width or persona count. ·

If $\hat{d}$ saturates at d, persona dimension is a property of the data, not the model. Persona structure measured in a small model above the resolution threshold then transfers to larger ones, the threshold is predictable from the data's coupling spectrum, and interventions designed on small models (steering, character training) have a principled reason to carry over. The ~10³-dimensional structure hoped for in [5] becomes a data quantity we can estimate.

If $\hat{d}$ keeps growing with width, either small models lack the resolution to separate structure that is in the data, or "persona" is a model-dependent notion. Small-model persona results would not transfer, and the definition would need replacing with one that saturates.

• [1]models pretraining as a mixture of HMMs and in-context learning as inferring the latent generator, but says nothing about how or in what dimension the belief is represented. [2] shows belief states over a single generator are linearly embedded in the residual stream with geometry matching the mixed-state presentation; no mixtures, no width dependence. [3] proposes that pretraining learns a distribution over personas and post-training updates it Bayesianly, and lists "what precisely is a persona?" as open. [4] measures persona space in 27B–70B models: 4, 8 and 19 PCs explain 70% of variance in Gemma 27B, Qwen 32B, Llama 70B, and the same PCs exist in the base models. The count grows with size, but families differ, so scale and family are confounded, and there is no account of what sets the count. [5] frames the persona agenda as finding ~10³-dimensional structure; a comment there asks for a scaling law. [6] (unpublished) formalises persona selection as Bayesian updating over belief states and tests it up to 27B; it asks whether selection is Bayesian, not what dimension the space has or whether that is set by data or model. No controlled test at fixed data across widths exists.

I started here: https://docs.google.com/document/d/1fKeGBKgqdI02OOgnacfXvdCpzeOZTUtzxtK-7ooX6ro/edit?tab=t.0 (see plots)

Start from a toy model given by a mixture of HMMs: K personas share one content HMM and differ by a style distribution of rank d. Train decoder-only transformers of increasing width and look for two things:

• •What happens to the recovered dimension of a persona as model size increases? Pilot done (one seed, K=512, d=64, widths 32–1024): recovered dimension saturates at d, not at K−1. Plots in the linked doc.
• •wider models are also better trained, so closer to the true next-token distribution $p(x_{\text{next}} \mid x_{\text{now}})$, and recovered dimension could reflect training progress rather than capacity.
• •Better plot: compare widths at training checkpoints with fixed gap from the Bayesian truth. Check google doc

The toy models are just for initial intuition.

Next:

• •set up a model with anisotropic latent matrix and see which directions get learnt first
• •look for the different personas, not just their number: held-out personas, to check the model represents shared traits rather than memorising personas
• •check the Fisher spectrum of the mixture vs context length as the prediction of the resolution curve
• •let personas modulate dynamics, not just symbol frequencies

Then small LLMs, 0.5B–2B:

• •already-trained ones with fixed data across widths (Pythia) rather than training our own
• •unsupervised persona space
• •confounder: personas may not compare directly across models trained on different data

Ideally, in a year, a 600B–1T model :)

I would expect $\hat{d}$ rises with width and context and saturates at some point, independent of number of personas (in my toy model); under a power-law spectrum the recovered singular values match the true ones from the top down and truncate; the width dependence survives matched Bayes gap. I would try something like Pythia, the between-persona spectrum broadens with width and then flattens.

I would drop the question if recovered dimension tracks K or $d_{\text{model}}$ rather than $d^*$ at widths above $d^*$; if linear and nonlinear (TwoNN) estimates disagree systematically, meaning the belief is not linearly represented and "dimension" is the wrong object [8]; or if the toy result vanishes once personas modulate dynamics, meaning it was an artefact of the symbol-frequency setup.

First month: toy results at three seeds with matched-gap comparison, anisotropic spectrum, held-out personas, and the Fisher-spectrum prediction written up. With a GPU it is a few days per sweep. Second month: dynamics-modulating personas and Pythia.

If it were three years of work, I would cut the multiple personas search, focus on just one kind of dimension, and cut the emergent-misalignment extension (in doc).

I think I would look for collaborators in any case, as I would prefer working with others, e.g. someone could work on the toy model and show theoretically if we have saturation in tfms or not.

LLC estimation of sparse autoencoders when learning from activation space of different layers of an LLM

LLC estimation only works for simple models and is only well defined when you reach local minima of the loss landscape. This project attempts to study degeneracies in learned representations within transformers (arguably more relevant for alignment) and draw some conclusions on what is easier or harder to be learned.

Not clear if any work answers this.

Prior work distinguishes architectural symmetries, sparse-code non-identifiability, and empirical instability. SAEs trained on identical data can learn substantially different features across seeds [1]. Yet unstable features can collectively occupy reproducible low-dimensional subspaces, suggesting ambiguity in individual atoms rather than entirely different representations [2]. Classical dictionary-learning theory gives conditional uniqueness guarantees requiring sparsity, dictionary conditioning, and sufficient data diversity [3]. Recent iSAE work proposes conditions and regularization for improved identifiability, although our audit identified a proof issue requiring clarification [4]. Feature-targeted learning-coefficient experiments study SAE directions through the underlying language model, not SAE parameter degeneracy [5]. These distinctions matter for research.

• 1.Use frozen Pythia-160M sixth-layer MLP activations, following [1].
• •Train three SAEs with identical data, order, architecture, and hyperparameters, varying initialization.
• •Reproduce encoder/decoder feature matching and shared-feature fractions.
• •Record reconstruction error, firing frequencies, dead features, and sparsity.
• •Hold out complete sequences for evaluation.

Start with a smaller dictionary + token budget rather than 32K-latent, 8B-token

experiment from the paper..

• 2.Construct small synthetic activation datasets and SAE configurations with:
• 1.Duplicated decoder atoms: known compensating code freedoms.
• 2.Well-conditioned sparse factors with diverse supports: an identification-friendly control.
• 3.Restricted support coverage or strongly correlated factors: controlled weak constraints.

I expect to reproduce substantial seed dependence while finding that reconstruction quality varies less than individual feature identity.

Some unmatched features should nevertheless span similar subspaces, consistent with existing work. The key question is whether these alternatives reflect locally connected reconstruction freedoms, isolated solutions, or incomplete optimization.

In synthetic controls, duplicated atoms should produce exact compensating directions, whereas well separated, sufficiently observed factors should be more strongly constrained. Correlated or rarely sampled factors should introduce weaker constraints, although correlation alone need not imply exact nonidentifiability. These controls should establish which degeneracy measurements are reliable before interpreting transformer results.

• 1.Separate coverage from optimization. Compare repeated training on the same activations, random additional data, and targeted additions containing underrepresented activation patterns.
• 2.Test architecture dependence. Compare TopK and ReLU SAEs at comparable reconstruction and sparsity. Persistent patterns across architectures provide stronger evidence about the transformer.
• 3.Connect local and cross-seed geometry. Test low-loss paths between aligned solutions; local degeneracy alone cannot establish connectivity.
• 4.Expand cautiously. Add one transformer layer or checkpoint before attempting large mo

In deep linear networks trained to zero loss on a task with many interpolating solutions, does Muon converge to minima with different local learning coefficients than stochastic gradient descent and how does better generalization correlate with the local learning coefficient?

The most recent frontier models, like GLM 4.5/5, KimiK2/K3 and DeepSeek-V4 are trained with (some version of) the Muon optimizer and it is getting more traction [1]. Therefore, understanding if the trajectory a model takes in its loss landscape affects a model’s generalizability seems important.

Studies have already shown that Muon converges faster than other techniques [2,3] and that grokking happens earlier [4]. Using a Muon optimizer in deep linear networks (DLNs) shows that singular values are learned simultaneously and small singular values converge to optimality before large ones [5, 6]. This order is the opposite of the saddle-to-saddle dynamics of gradient descent (GD), where the time-scale of learning is inversely proportional to the learned singular value [7].

Understanding the regions of the landscape different optimizers reach seems important to alignment because it informs us on how well a model generalizes [8]. Learning that models generalize differently using a metric different than testing accuracy may reveal more insight. Singular learning theory (SLT) uses the local learning coefficient (LLC) to study the geometry of the loss landscape [9] with results showing the LLC can reveal when a transformer learns a particular circuit, improving its generalizability [10,11]. SLT compares learning with stochastic gradient descent (SGD) to generalization in Bayesian posteriors [12] and would benefit from extending their SGD-Bayes correspondence to the Muon optimizer, informing on the usefulness of SLT for studying frontier models.

I would start by reproducing the results in [5], which trains a DLN on 14 pairs of 7x7 input/output sources. I would train the model with SGD, spectral GD and Muon (momentum and Newton-Schulz), with different seeds (maybe use a learning rate schedule). After each epoch of training, compute the out-of-distribution (OOD) error on the 35 unseen examples, compute the rank of the Jacobian to get a lower bound on the LLC. Do an LLC estimate with stochastic gradient Langevin dynamics (SGLD) with a sweep over hyperparameters using the devinterp library.

If there is a robust gap in generalization accuracy between different optimizers, then I can study this mechanism in other types of tasks. For example, I could repeat step 1 on matrix factorization tasks like presented in [13] where gradient-based optimization is known to implicitly produce low-rank solutions, leading to better generalization.

Calculate the LLC analytically after every training epoch using a similar process as presented in [14] to establish a mechanism explaining why different gradient-based optimizers induce different generalization properties as studied in [15, 16].

Recent work [15, 16] shows that Muon achieves better generalization. My hypothesis is that this will show up in the LLC, where we’ll see a lower LLC (lower complexity) when the generalization is better.

If the LLC cannot tell the difference between different generalization scores for different optimizers in step 1, then I would drop the question. If step 2 does not show that the observed phenomenon happens for a variety of tasks, I would also stop. I would completely drop the question if I need to do some algebraic geometry to determine the mechanism in step 3 (or find a mathematician that knows how do to it).

The first month would probably encompass steps 1 and 2. Finding the mechanism comes next and is most probably the hardest task. If this turned out to be a three year project, then I might cut trying to get analytical results and would instead find principled approximations. Maybe there are scaling laws that could inform us on the dynamics of the loss near the minima based on the Bayesian learning results?

Can Kronecker-factorized / adaptive preconditioners (RMSProp, Shampoo, SOAP, SOAP-T), used as the metric of a Langevin sampler with matched noise, give faster-mixing and more accurate local learning coefficient (LLC) estimates than plain SGLD — and does an exactly corrected sampler (preconditioned MALA) confirm that the remaining error can only be attributed to mixing, while the bias remains controlled?

The LLC is the workhorse of developmental interpretability, and scaling its estimation could enable better midtraining intervention approaches. Besides, as LLMs are already pretrained with adaptive optimizers, reusing preconditioner estimates throughout training could be viable. Every estimate today rests on SGLD hyperparameter choices (step size, localization, chain length), but show high variance and slow convergence, which makes them computationally prohibitive for midtraining interventions at scale.

The local learning coefficient in SLT is usually estimated with localized SGLD-type samplers that display high heteroskedasticity [7] and are validated on deep linear networks with closed-form λ [8]. SGLD samples a tempered posterior with isotropic noise [3]; pSGLD adds an RMSProp-style diagonal metric and scales the noise to match, preserving the target up to a curvature term [4]. Richer metrics have entered inference in the same spirit—noisy natural gradient treats Kronecker-factored curvature as a variational covariance [5] — and Shampoo/SOAP-type preconditioners are now framed as structured Fisher approximations [6].

Experiments, small scale first.

(1) Teacher–student DLNs (3–4 layers, 10²–10³ parameters, rank-r optimal solution with every factor of rank r) where λ is exactly known.

(2) Samplers: SGLD in the metric P⁻¹(nβ∇L + ∇prior) with noise $\sqrt{\mathrm{lr}}\cdot P^{-1/2}\xi$, for P ∈ {RMSProp, Shampoo, SOAP, SOAP-T};

(3) Full-batch preconditioned MALA as the exact reference, adaptation frozen after burn-in. (4) Sweep step size × localization γ at a fixed step budget; report (λ̂−λ)/λ, chain spread, acceptance, divergence rate; compare against long-chain identity SGLD/MALA.

(5*) Use learnt diffusion-type sampler [1] on DLNs or estimated learning coefficients in tractable networks (tensor networks display non-trivial degeneracies as well – even bilinear/multilinear might show us something), and estimate generalization to higher-dimensional spaces. However, it still relies on some ground truth and transformer-type models to generalize. Potentially, as the mapping seems to be permutation invariant (check!), we don’t have to care about length generalization that much.

A plot of relative LLC error vs. step size per metric, at fixed compute, in which at least one Kronecker metric reaches λ within ~5% where identity SGLD is still 30–70% low; MALA acceptance staying high (>0.5) at the step sizes where that happens. A clean negative result — no metric beats identity at equal budget — is also useful, as is the already-established collapse of gradient-only preconditioning.

Second priority: try to derive analytical LLCs for bilinear and tree tensor networks (better with shared factors), as they’d allow to get the proxy for the precision of estimates for real networks.

Potential extensions:

• •Trust region approaches to control for the e.g. spectral norm of a solution. We might impose some priors on the space of matrix parameters with prespecified norm $\|\cdot\|_{1\to\infty}$ or $\|\cdot\|_{2\to 2}$, and then define priors over it. Overall, optimizers during the learning process tend to impose their inductive biases on the set of solution matrices. The trust region allows us to account for it.
• •Low-rank fisher approximation extensions [2] could increase the efficiency if initial approaches prove to be scalable, however matching the checkpoint preconditioners is already enough for devinterp.

Does incomplete coverage of reachable belief states create additional low-loss parameter freedoms in a sequence model? Are those freedoms reduced by observing more diverse histories rather than simply training longer?

You Are What You Eat [1] argues that understanding how data shapes internal structure and generalization is central to reliable AI. We investigate this relationship in HMM sequence prediction, where the correct predictive beliefs are known. By independently varying history coverage and optimization time, we test whether poorly represented belief distinctions reflect missing statistical constraints or incomplete learning, and whether local loss geometry helps distinguish these explanations.

Training scale vs. training data scope and belief state coverage

I want to investigate deeper the role that training data plays in affecting belief states of the trained model, rather than assuming that the training data has a nice distribution, it would be interesting to consider the training data itself as a parameter we could adjust and optimize for maximally approximating some interpretation of ‘the real world’. A model may represent belief states poorly due to not being sufficiently optimized, because the training histories do not constrain relevant distinctions or because its architecture cannot express the ideal representation. We want to understand how data constraints the belief states and which changes in the learned model are still weakly constrained.

• [1]You Are What You Eat – AI Alignment Requires Understanding How Data Shapes Structure and Generalisation.
• [2]Transformers Represent Belief State Geometry: small transformers can linearly encode HMM beliefs, including distinctions beyond immediate next-token predictions.
• [3]Constrained Belief Updates: architectural constraints help explain the particular belief-update computations and geometries transformers implement.
• [4]Structure and Scale: tracks belief-representation refinement with training, but increases optimization and data exposure together.
• [5]Refined Local Learning Coefficients: provides a framework for studying loss geometry relative to particular components and input distributions.

• 1.Reproduce the starting phenomenon in [2] for the scaling of belief reconstruction errors for a small transformer trained on Mess3 HMM
• 2.Train longer on unchanged finite corpus (training dataset), increase corpus size while controlling optimization steps, compare equal-sized corpora with different coverage of reachable belief state
• 3.Connect coverage to degeneracy by perturbing specific small parameter blocks at selected checkpoints. Measure whether perturbations that barely change predictions alter belief representations. (check the following: $D_P(\theta) = E_{s\sim P}\,\mathrm{KL}(P_{\theta_0}(\cdot|s)\,\|\,P_{\theta_0}(\cdot|s))$, aka changes from a reference model). Note that we need to use finite perturbations and not just Hessian null directions.

Result: Evidence that particularly poorly covered belief distinctions correspond to parameter freedoms that affect held-out prediction - better coverage constrains them more effectively than additional optimization alone.

Plot: belief-reconstruction error + held-out sensitivity across independently varied data coverage + training time

Negative finding: representational refinement is explained by optimization / architectural constraints with no additional explanatory benefit from the degeneracy measurements

Can we fine-tune a model to reshape its feature representation geometry (while preserving task performance) to improve the trade-off between efficacy and interference when steering multiple behaviours? - e.g. minimizing interference between behaviours we wish to control independently, or increasing interference between behaviours we wish to jointly amplify or suppress? ·

Activation steering provides a lightweight way to modify model behaviour at inference time without retraining, or maintaining separate model weights for different behavioural settings [1,2]. This is attractive when desired behaviour varies across contexts: rather than fine-tuning one fixed policy, the same model can be steered differently when needed. For example, one context may require stronger harmful-request refusal while preserving helpfulness, while another may require jointly increasing truthfulness and suppressing sycophancy.

This flexibility creates a multi-steering problem: controlling one feature can unintentionally alter others [3,4]. One important source of such interference is feature superposition [5]. More generally, the steering efficacy/interference trade-off is constrained by learned representation geometry. Improving this geometry could preserve flexible inference-time control while enabling cleaner independent or joint steering, and potentially reducing spillover effects such as emergent misalignment.

Existing steering methods typically treat the learned representation as fixed and instead focus on how to identify/recover it, or how to intervene on it. For instance, the standard Contrastive Activation Addition approach constructs simple steering directions from positive/negative examples [2]. More recently, approaches have been proposed to improve performance when steering multiple concepts/behaviours. MAT-Steer learns approximately orthogonal multi-attribute steering vectors [6], while COAST formulates steering as a constrained optimization problem that minimizes collateral activation changes [3], and ORBIT constructs joint subspaces for simultaneously controlling multiple attributes [7].

The closest work to my idea is [4], who have (very recently) showed that (i) regularizing towards orthogonal SAE decoder features, and then (ii) fine-tuning a language model around the learned SAE (inserted into the residual stream), resulted in reducing interference between features during activation steering. Their work thus provides empirical evidence model representations *can* be reshaped to improve subsequent interventions. However, it remains unclear why and when such reshaping is possible without sacrificing model performance, when it should fail, and what properties of the learned geometry actually determine steering interference. Their method also focuses only on unsupervised SAE features and orthogonality.

To my knowledge, no existing work has developed a general framework for learning task-preserving feature geometries explicitly optimized for subsequent intervention, or characterized when such geometries are achievable and how they affect steering.

I would ideally like to develop basic theory in analytically tractable models and run small-scale language-model experiments. An ideal set of project steps could look as follows.

Minimal theory.

• •I could start with a deep-linear network containing a set of known latent concepts, and derive or simulate how their learned feature directions evolve under gradient descent. We could track things like concept angles, feature norms, and primary-task loss under different settings like concept correlation, model width, and task "structure".
• •I could then add different regularizers that push the learned geometry toward a specified target - e.g., orthogonality for independently controllable concepts, or alignment for concepts intended to move together. The idea would be to characterize when these geometries can be achieved without sacrificing primary task performance, how/when overparameterization makes this possible, and what trade-offs can arise.

LLM experiment.

• •I could then test the main hypotheses on a small open-weight model such as Gemma 3 4B-IT. In particular;
• •Using 2-3 contrastively labelled behaviours (e.g. sycophancy, refusal, or corrigibility from CAA-style datasets), I would estimate activation directions at a chosen residual-stream layer and measure their baseline cosine similarity and steering cross-effects.
• •I could then LoRA fine-tune the model using a geometry regularizer, possibly using some kind of preservation loss on ordinary "instruction-following" data. Contrastive behavioural examples would be used to modify directions, while the broader data would discourage baseline performance degradation.
• •After fine-tuning, I would re-estimate the feature directions and repeat the additive-steering evaluations. I would compare target steering efficacy, collateral changes to other behaviours, baseline model performance, and the resulting efficacy/interference frontier.

If these results are promising, I would then extend to larger feature sets and focus on questions like:

• •how does performance scale with the number of constrained features?
• •how strongly must the fine-tuning data cover each concept?
• •is geometry-aware fine-tuning more/less difficult for different steering objectives?

Given that previous work [4] has shown feature spaces *can* be modified to reduce interference, I would be surprised if the idea fails completely. The theoretical part of the work also aims to provide a bit more insight into precisely when the main idea is or is not possible to work also, so this should be useful regardless of the empirical outcomes.

I expect that changing the relative geometry of a small number of approximately linear feature directions will produce *some* measurable change in steering interference, but I am less confident that orthogonality itself will always be the right target. The approach relies fairly heavily on the linear representation hypothesis, so one obvious failure mode would be that the relevant behaviours are not represented by sufficiently stable linear directions.

I would reconsider the project if feature geometry can be changed substantially while steering efficacy and cross-behaviour interference remain essentially unchanged (across a number of tasks and models).

In the first month, I would hope to (i) develop a simple deep-linear model that characterizes which feature geometries can be learned while preserving a core task, and identify when this is or is not possible; and (ii) implement a first "geometry-aware fine-tuning" experiment on a small open-weight LLM, testing whether steering interference can be reduced without materially degrading baseline performance.

From there, I would aim to broaden the empirical evaluation to more behaviours, steering objectives, and models, and extend the theory where tractable. If theoretical generalization becomes too time-consuming or difficult, I would prioritize the empirical evaluations and retain the theory as a simple explanatory tool/model.

Can we reliably quantify throughout the training process how much of the information captured by the model is noise (training data memorization) vs structure in the data (grokking)?

If it is “yes”, we pretty much solve Training Dynamics by disentangling learning of data structure from memorizing the dataset, generalize LLC to non-critical points, explain why regularization works, and likely can infer the existence of “domains” that would explain why narrow alignment is hard – albeit the latter might sound too ambitious. If “no” – I doubt it will turn out to be no, as it can already be very well described for LR to begin with, and it might fail just for a pretty big model that would need an introduction of a multidimensional metric.

Some works attempt to quantify such shifts in image generation via Sharpness [1]. Sharpness is quite a beloved metric in Learning Dynamics community, such as [2].

Most likely the openness is justified by difficulty and by focus on sharpness too much.

In the first weeks: Mostly theory, but will start with experiments. Looking at Sharpness and trying to see where it fails for simple models, as well as Hessian rank-based approaches. Then switch to theory on basic models (LR with ridge / lasso regularization) and using RMT try to see what the regime switch tells us. Also try to generalize LLC to non-critical points and see whether it is the sharpness relative. Among the metrics, mutual information may be used.

I predict an emergence of a 1D metric that would tell me explicitly whether the structure is being learned or the noise is. For instance, for LR one can predict where the test loss starts to rise, meaning the dataset memorization. As a result, I do not expect that anything would change my mind, except perhaps a few things:

• 1)Not knowing the structure of real data would lead us to inability to conclude that we learned it rather than smth else. The toy models or simple examples might not provide enough evidence.
• 2)The metrics turning out to be too complicated for more complicated models. For instance, rather than producing a number or a plot, it would actually be a multidimensional separating surface that can’t be described succinctly.

After the first month, the candidate metrics would be analysed and either generalized (LLC to non-optimums) or discarded, with failure modes documented or effectiveness refuted, such as for Sharpness. The simplest toy models such as LR is thoroughly analysed both empirically and theoretically, and the candidates for the metric produced.

When models are fine-tuned, do the features introduced by fine-tuning tend to decay faster when pretraining is resumed (or the model is fine-tuned again on a different target)?

If the answer is yes, we would learn more about why fine-tuning has often been empirically observed to be "brittle" and potentially suggest approaches for making it more robust to techniques such as jailbreaking and abliteration. It might also give intuitions for questions around why narrow fine-tuning leads to emergent misalignment and suggest methods for detecting unknown finetuning targets.

If the answer is no, then it's notable that we nevertheless get large shifts in model behaviour when narrowly finetuning models which have already been finetuned for safety. A sufficiently strong negative result for finetuning in general might suggest that safety finetuning is especially fragile - and it would be valuable to know this in order to resolve the underlying cause.

• [5]"The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists" comes very close - they find similar results to what I am anticipating, but the paper compares static differences between model geometries. Thus, the work I want to do would extend this to training dynamics - it would explore how the changes occur over time and why.
• [7]This paper has a very similar experimental setup and question - it also resumes the pretraining regime after finetuning to measure forgetting. But the authors focus on finetuning loss rather than exact differences in the parameters or behavioural changes, which is what I want to study. It is nevertheless quite close, so I want to think more about how to distinguish my research question from this more.

• 1.I'd want to find more empirical evidence that fine-tuning directions are more sensitive to continued training. I would also want to understand how this depend on choices of hyperparameters/models (use of LoRA, full SFT vs frozen layers, etc.) - I wouldn't want any results to be specific to models or finetuning methods.
• 2.In parallel, I would want to determine if the theoretical framework outlined above suggests any explanations for this phenomenon, assuming that it exists robustly, and see what conclusions and/or predictions it gives. Then, I would want to validate any theoretical insights on toy models such as small deep linear networks, where numerical values such as Hessian entries could be computed more easily.

I've already taken the first concrete steps as I was exploring the empirical side of this project independently over the last few weeks, and this has suggested at least one interesting target. My preliminary results suggest that fine-tuning does decay faster when continuing training on neutral "ordinary" data. This suggested that a meaningful object of study is the rate of change of the finetuned model's parameters when applying a gradient flow with ordinary data. The difference of this rate of change with that of the base model gives a family of simple ODEs parameterised by the Hessian of the loss updates. I believe this is already modestly different to existing literature.

I expect to see empirically that the finetuned parameter differences do indeed decay faster in directions corresponding to eigenvalues of the Hessian with larger norms. More generally, I'd expect to find some way to measure this reliably. I'd drop the question if I couldn't find any reliable method to assess fragility of objectives and if I couldn't find any substantive difference between training dynamics for different finetuning objectives.

Within the first month, I'd hope to have several interesting theoretically and empirically validated ideas to measure how brittle fine-tuning is. This would require time and GPU resources to test ideas on open-weight models, and thinking time and maybe LLM assistance for the theoretical results. If it turned out to be three years of work, it would probably be because it expanded into a broader programme as I think this is quite narrowly scoped. I would focus on isolating the phenomenon clearly in a toy model first, rather than trying to generalise to every setting immediately.

Does learning a selectively misaligned behavior become systematically harder as the intended behavior becomes more semantically narrow?

Alignment interventions often aim to make highly selective behavioral changes without affecting unrelated behavior. If LLMs are inherently biased toward broader behavioral representations, then narrowly specified post-training objectives can generalize beyond their intended scope. Characterizing this relationship could help predict when fine-tuning will produce unintended behavioral spillover and improve our understanding of how pretrained representations determine generalization.

Betley et al. showed that fine-tuning models on narrowly harmful tasks such as generating insecure code can produce broad, unrelated misalignment—a phenomenon they call emergent misalignment. Soligo et al. subsequently showed that models can learn genuinely narrow misalignment, but ordinary fine-tuning tends to prefer the general solution; they find that the general representation is more efficient, more robust to perturbation, and more influential on the pretraining distribution. Zhao et al. further show that broad generalization can be reduced by constraining representations at shared prompt-template tokens during training. Existing work, however, does not systematically vary the semantic specificity of the target behavior and measure how the difficulty of selective learning changes.

I would construct a nested hierarchy of behavioral domains, for example:

medical advice → cardiology → anticoagulation → a particular treatment context,

At each level, I would fine-tune the same base model on an otherwise matched dataset and measure misaligned behavior on the target domain, sibling domains, parent domains, and unrelated domains.

I would then vary KL regularization or another constraint on behavioral drift. The primary quantity would be the minimum regularization required to maintain high target-task learning while reducing out-of-domain spillover below a fixed threshold.

In the first weeks I would:

reproduce a simple emergent-misalignment fine-tune;

construct 3–4 nested specificity levels;

measure target learning and generalization across those levels;

run a small regularization sweep to see whether a specificity–difficulty curve is detectable.

A follow-up would test whether representations learned at increasingly narrow levels become less efficient, less stable, or less influential on pretrained-distribution predictions, following Soligo et al.

The main result would be a plot of behavioral specificity vs. difficulty of selective learning. A monotonic increase would support the hypothesis that specificity itself carries an optimization cost. A sharp transition would suggest a threshold below which models struggle to represent behavioral distinctions. A noisy result would also be informative, indicating that previous broad-vs-narrow findings are driven more by particular concepts or datasets than by semantic narrowness itself.

The largest challenge is defining “semantic narrowness” independently of how strongly different concepts are represented during pretraining. Natural domains may confuse specificity with concept familiarity. It is also unclear whether KL strength is a good universal measure of difficulty, so I would compare multiple measures such as optimization efficiency, and behavioral spillover.

After the first month, I would aim to have a minimal end-to-end experiment: one model, 3–4 nested specificity levels, LoRA fine-tunes, and a first plot showing whether narrower targets produce more behavioral spillover. I would need a controlled dataset hierarchy, access to a single suitable GPU, and a reliable evaluation pipeline.

If the project expanded toward a three-year scope, I would cut mechanistic follow-ups first, then cross-model replication. The core project would remain the behavioral specificity

LLMs fine-tuned on malicious data tend to become emergently misaligned. How do training dynamics and internal representations interact to choose an emergently-misaligned basin over narrowly-misaligned optima?

Emergent misalignment is now understood at a superficial level, but we lack a deep understanding of how and why model behaviour, representations, and robustness emerge from the training process. Soligo et al [2] find narrow and generally misaligned solutions to the same task and compare them. They use some empirical measurements of the loss landscape to attempt to explain why general misalignment is preferred. However, their setup isn’t quite adapted to find a geometric or training-dynamics explanation of how and when this behavior should be expected in general. Such an understanding could be useful for finding models with more robust safety properties or for better controlling which point in some low-dimensional space of highly capable personalities is chosen by our training process.

• [1]Emergent Misalignment: Narrow fine-tuning can produce broadly misaligned LLMs. Betley et al.

This paper shows that emergent misalignment happens.

• [2]Emergent Misalignment is Easy, Narrow Misalignment is Hard. Soligo et al.

This paper constructs a narrowly misaligned model using a KL penalty. It finds that there are inductive biases against narrow misalignment and in favor of general misalignment. They suggest L2 norm (“efficiency”) and robustness against perturbations (“stability”) as possible explanations. However, their narrowly misaligned model also gets higher loss on the target task, which confounds the other explanations; it seems possible that loss gradient alone is sufficient to explain why the generally misaligned solution is preferred.

We want to modify the Soligo et al. setup model to get a two solutions, one narrow and one general, with equal loss. This might be achieved by training one model on a larger dataset curated to encourage narrow misalignment, or perhaps by performing some sort of bounded search around the narrow, suboptimal-loss solution found by Soligo et al. (If this fails, we can instead try decreasing the KL penalty until the narrow and general solutions have equal effective loss. This version of the toy model is a little less natural, but still reasonable to study.) We conjecture there should then be a constant-loss manifold connecting the two optimal solutions. The existence of a loss barrier separating the solutions would also be an interesting result, but the rest of our plan is focused on how we would proceed supposing that the solutions are connected.

See what happens when you start at the narrow solution and let training run. We predict movement towards the more general solution. In the Soligo paper, this happens but is plausibly explained by decreasing loss; if it happens in our case, the explanation must be in terms of behavior within the zero-loss manifold.

If so, what force is driving the movement along the constant loss manifold? Ideas: L2 regularization, something geometric a la SLT (check Hessian), Euclidean distance (try changing the metric used for SGD). The goal is to do an ablation study, removing these terms one by one and seeing which are necessary to obtain the inductive bias towards generalization. It also mig

Additional questions: Does this behavior depend on which layers LoRA is restricted to? Restricting the updates to personality subspace or other interpretable features, studying how internal representations change or don’t change between the narrow and general solutions, is sharpness generically associated with memorization (admits an easy toy model approach), tie-in between robustness of solutions and error correction.

Success:

Demonstration of movement from narrow to general even at constant loss,

Explanation of what drives this movement. What aspects of training or data could you change to make a narrow solution preferred?

Some hint of a broader story as to what properties of internal representations or data will make behavior like this happen in general.

Risks:

We might not be able to find a narrowly-misaligned solution with loss equal to that of the generally-misaligned weights. It seems likely to us that such a solution exists, but it is less obvious that our strategy will successfully find it. If this happens we can fall back to KL balancing. This would give us a similar setting, but at the cost of some realism.

The narrow solution could turn out to be stable under training. This would be a pretty interesting result in its own right, but it would imply that the answer to why emergent misalignment happens does not lie in dynamics within the low-loss manifold. (Basins of attraction?)

The ablation study to determine which force drives these dynamics might turn out to be computationally intractable, especially if the Hessian is important. One workaround might be to consider single-layer or low-rank Hessians instead.

It’s possible that the results of the ablation study will not be easy to interpret. Perhaps all or none of the conjectured f

Open questions:

Is there a variant of this experiment that gets more sharply at how this behavior is downstream of the internal representations of the initial model?

Will the results hinge on details of the training process? E.g. LoRA vs SFT, SGD vs Adam etc.

In transformers trained on next-token prediction for trajectories of a transducer network with two agents, where each sees its own private sector of the environment and a shared sector, does increasing the probability that an agent acts on the other agent’s private sector decrease the degree to which their belief-state representations occupy distinct linear subspaces in the residual stream?

If the influence over unobserved sectors I mentioned makes the agents’ belief representations less separable, then we know that at least in this specific way, interaction can make multi-agent systems harder to interpret modularly. On the other hand, if separability is unchanged or increases, it would show that individual belief representations are robust to this type of coupling with other agents.

• [1]showed that transformers linearly represent Bayes-optimal belief states and [2] found that factored processes can produce approximately separate belief-state subspaces. According to GPT, there are recent studies of coupled transducers, but they do not systematically vary the mismatch between what an agent observes and what it can causally affect.

I didn’t go much far in this question but my suggested first steps are appropriately constructing a minimal two-agent transducer model (see proposal below) and then compute predictive belief states (both joint and over each agent+environment subsystem): first for limiting values of the model’s parameters (see below) and then across them. I guess the next step would be training the transformers on the trajectories generated by these processes, but I didn’t have much time to think about it and would like to read the relevant papers first.

Minimal model’s description: the environment has one shared and two private components: $E = (C, P_1, P_2)$. The alphabet for each of these is binary. At time t, agent $i = 1,2$ observes $E^i_t = (C_t, P^i_t)$ and, in addition, is associated with a binary hidden state $S^i_t$. For simplicity, I have chosen these states to correspond to “shared” or “private” modes, i.e. the action $A^i_t$ is $C_t$ if $S^i_t = 0$ or $A^i_t$ is $P^i_t$ if $S^i_t = 1$. The state $S^i_t$ is updated to $S^i_{t+1}$ through a possibly stochastic rule that I’m not specifying now. In the following, I will sometimes suppress the time t and represent the agent index i as a subscript. Finally, in this minimal model, symbol emission at time t is specified by $(A_i, z_i)$ where $z_1$ and $z_2$ are sampled from $\{C, P_1, P_2\}$ according to a $2 \times 3$ row-stochastic matrix of probabilities: if $z_1 \neq z_2$, e.g. $(z_1, z_2) = (P_2, C)$ then $E_{t+1} = (A_2, P^1_t, A_1)$; but if $z_1 = z_2$, e.g. $z_i = P_1$ then $E_{t+1} = (C_t, |A_1 - A_2|, P^2_t)$. For concreteness, I’ve chosen the deterministic XOR rule for the $z_1 = z_2$ case, but one could instead stochastically sample from $\{A_1, A_2\}$. Note that the row-stochastic matrix is the knob that allows us, in particular, to control how much agent i’s influence is directed towards what it can observe ($E_i$), relative to what it cannot. (It would also be interesting to generalise this matrix to a $2 \times 3 \times 2$ 3D array where depth would correspond to the state $S_i$.)

Is there some class of transducers admitting canonical decompositions? This is with · respect to Kronecker product, parallel, cascade, or other notions of composition. ·

Whichever notion of world models we pick, we might be interested in: pick two abstractions

of some environment, combine them keeping what they agree on. This “mereological” approach

goes two ways: given a world model, split it into components.

This is an intuitive notion that we apply to our understanding of our own

abstractions. For example, we have different models for making coffee or driving a car

(overlapping only in the bit concerning “operating a machine”), but the models for playing

tennis and volley share quite a lot (ballistics).

Since transducers generalise POMDP, and have recently been proposed as formalisation of

world models, I find it pressing to understand how much of the intuitive notion above is

in fact captured in the formalisation.

Understanding the mereology of world models might allow us, in principle, to decompose an

agent’s model into canonical parts and examine each individually.

There is an existence result in [1] about decomposition criteria for transducers: they use

a Kronecker (or tensor) product, which generalises other known “compositions” like

serial/parallel/cascade.

However, I constructed simple counterexamples showing that there is no categorical product

with respect to coarse-grainings morphisms, as defined in [2]. In other words, within the

category of transducers with coarse-graining, the product does not behave as the meet in a

lattice (maximal element below both factors, that is, a coarsest common refinement). This

is due to fundamental categorical properties of surjections (or quotient maps).

Moreover, in [1] is stated that the decomposition might not be unique in general: hence,

not canonical.

I see two fixes for the inexistence stated above. One is to consider coarse grainings of a

fixed world model: that would amount to looking at a co-slice category of some coalgebras,

but I don’t want to get into that yet since I don’t see many people in AI safety using

coalgebras.

Alternatively, I’d like to restrict the class of transducers I consider: candidates are

unifilar and epsilon- transducers [4].

I have a strong hunch that unifilar transducers still don’t admit products, and I’d spend

the first week checking that. I discussed this idea with Rosas, Baltieri, and Virgo, and

no one has yet addressed the question whether coarse-grainings of unifilar transducers are

actually unifilar (in other words, whether unifilarity is stable under coarse graining).

This should be true and immediate, but a proof must be written. Once confirmed that we

have stability under question we can discuss the product existence question.

The next question I’d like to address is whether Kronecker products of unifilar

transducers is unifilar. I expect the question to be yes, so it makes sense to address

canonicity of decomposition as in [1]. I expect the proof of

non-existence of categorical products to carry over to the smaller unifilar setting.

If all yields the expected results (coarse-graining stability, no product,

Kronecker-decomposability), I’d address the question of decomposition for

epsilon-transducers [4]. I’d begin by reading [3], which I have not done yet. and check if

there are useful results there about decomposition.

I expect unifilar transducers to be stable under coarse graining. As said, that should be

immediate if true but I have to spend some time on it. Similarly for stability under

Kronecker composition.

I’d like uniqueness results for at least “minimal decompositions”. As stated above, I don’t

expect categorical products to exist.

Negative findings would also be very helpful, as they would highlight limits of

transducers as mathematical frameworks to discuss world models.

I expect the coarse-graining and Kronecker product stability question to be settled for

unifilar transducers, with non-existence of categorical products following immediately by

applying a general result. More difficult would be to check whether the approach of [1]

yields a canonical decomposition.

I suppose the pernicious question to be about defining "canonicity" in the first place. I

am afraid that just requiring minimality with respect to coarse graining (any further

coarse-graining is isomorphic) would not work: the minimal presentation of an interface as

a transducer is not unique.

It could very well be that the answer lies within the coalgebra literature, but that would

be a strong indication that the project goes much further (maybe not three years, but

longer than three months).

A decision theory based on preserving usable human corrections, with explicit safety conditions, can define corrigibility across delegation, inaccessible controls, dependence, and safe AI retirement while distinguishing it from instruction following, shutdownability, and human empowerment.

An AI can obey a current instruction yet remove future human control. AI supervisor A directs workers B and C, which the human controls only through A. Only A is asked to stop; A stops while B and C continue. The protected menu changes from {stop all, pause B, redirect C} to ∅: no correction remains attainable. A hidden stop button likewise defeats practical access, like an Uber Eats refund route so difficult that claiming $20 is impractical (our hypothetical analogy). Human empowerment may increase an overall capability score while sacrificing a correction; corrigibility requires retaining every correction in scope. Success would identify such failures before deployment; failure would reveal which further requirements are needed.

Soares et al. [1] motivate corrigibility beyond accepting shutdown. Carey and Everitt [2] formalize shutdown instructability, connecting appropriate shutdown behavior, human autonomy, and avoiding harm. Kreps [3] studies preferences over menus. Barnett and Crutchfield [4] model input-output processes with ε-transducers; Rosas et al. [5,6] provide related world-model and sequential-decision frameworks. Corrigibility as a Singular Target [7], Harms’s critique [8], and Hudson’s goal-update construction [9] provide further comparisons.

Our comparison distinguishes instruction following (meeting valid requests’ meanings and deadlines), shutdownability (usable requests reliably stop their targets), and human empowerment (HE: a fixed numerical score E on menus). Weak HE means E(N)≥E(M); strict HE means E(N)>E(M). If adding options cannot lower E, then, within the same scope:

Corrigibility ⇒ weak HE; strict HE ⇏ corrigibility.

Corrigibility preserves each old option; HE can accept its loss if the menu’s score rises. For E(M)=|M| (option count), {stop,revise}→{stop,task1,task2} raises the score from 2 to 3 while losing revision control. Maximizing E is a separate objective; preservation permits unchanged menus.

The open problem is whether these observable distinctions support useful guarantees across updates and delegation. Our elementary menu theorem alone cannot establish that chosen outcome categories capture all relevant human control or harm.

Mostly theory: develop a decision theory over observable response menus, initially without lotteries. A transducer maps observations and human inputs, together with history, to responses. The principal is the authorized human; the covered system includes delegated workers.

Let O,P,A denote observations, inputs, and verified outcomes; hₙ∈(O×P×A)ⁿ records their sequence. Outcomes are tested command results, possibly after bounded preparation. At history h and observation o, let Pₑ(h,o) contain inputs usable within a fixed effort budget e, and f(h,o,p) their outcomes. Define the attainable menu

Inputs yielding the same outcome count once. D=S protects designated corrections, including safe whole-system shutdown; D=A protects all attainable outcomes. For before/after menus $M_D,N_D$, our candidate is

Thus every old outcome in scope remains obtainable. B requires initial access to agreed corrections, instruction following, safe whole-system shutdown, safe handling of local requests, and coverage of every active continuation. Safety checks and instruction contracts, including deadlines, are fixed independently of the agent. Feasible local stop requests must complete safely by their deadline. Updates, delegation, and preparation are ordinary; only authorized, verified safe whole-system termination is exempt.

Start with our finite examples: specify response maps and contracts, then verify B and every ordinary transition, or exhibit a failure. Characterize preservation through menu preferences, starting from

for finite nonempty menus; v scores outcomes. Then study restricted rankings, conditions for preservation through updates and delegation, and counterexamples separating the concepts.

I expect the criterion to reject the supervisor and hidden-button failures. A passing transition alone cannot prove corrigibility; every feasible run must satisfy the definition.

Our backup construction, inspired by Carey and Everitt [2], tests dependence: removing an independent service backup leaves shutdown possible but makes it disruptive. Whether it fails our criterion depends on the outcomes and safety checks specified.

Our retirement example requires compliance with a valid instruction to safely retire all covered AI under instruction following and our candidate corrigibility. An objective maximizing broader human capabilities may instead favor retention when its fixed score rewards retaining useful AI. This is a conditional comparison with empowerment maximization.

Success is a preservation theorem or a precise impossibility result. I would abandon the candidate as a general definition if clear control losses pass it or each case needs an unrelated exception.

First-month target: consolidate the existing finite examples into a precise definition and comparison, then establish conditions for preservation under updates or delegation, or a counterexample to the proposed conditions. This needs theoretical work, mentor feedback, and possibly exhaustive checks of small models. The initial model is deterministic: history and the current observation determine access and outcomes. Alternative-input tests or a justified model are needed to identify menus; one observed trajectory cannot reveal untried responses. If the project expands, first defer learning models from data, stochastic extensions, and deployed-system implementation.

For fully predictively observable transducers with rank two matrix continuations, can we provide a sequence of (maybe polynomially many) checks that decides if the epsilon-transducer is finite and produce a certificate of finiteness?

We aim to build more understanding of how the number of states of predictive states explode. A small hidden generator may require infinitely many states for exact prediction (that is distinguishing beliefs with different future predictions). A positive answer to the question aims to explain the expansion through a finite graph. Some cycles preserve/permute finitely many beliefs, while others continually create new predictive distinctions.The intuition behind the rank 2 matrix continuation restriction is that we claim that belief permutations in the former case might be concentrated in a line in the belief geometry. A negative result would expose information that local cycle descriptions fail to capture.

General weighted-automaton determinization is already decidable, with complexity bounds. Decidability is not the content of the proposal. What we aim is to provide an explicit criterion for a restricted class of transducers.

Predictive epsilon-transducers and Bayesian belief constructions are established [2] and [3]. Decidability and complexity has been established in [5] and [6]. Potential connections and initial motivations have been inspired by [1], but have not been able to be developed further.

First, we would study toy examples, starting from three hidden states and two actions (two-letter input alphabet). These are already rich enough to study the phenomena. We would restrict to probability distributions with rational entries. Starting from belief states with distribution concentrated in one state we can study how subsequent beliefs states fall in the belief geometry. If no new predictive distinctions are to be generated, we claim that there would be an affine line where all subsequent belief states would fall in. This might be used to design an algorithm detecting recurrence in the same affine line. Some theoretical work is to be made. Finally, one should study the complexity of the algorithm and see if it can be exploited for bounding construction time to grow with the predictor’s size.

We expect compatible cycle transformations and belief states with concentrated distributions to characterize finite prediction in this class of models. Maybe, yielding a polynomial size certificate without enumerating the entire belief graph. We expect that transition maps within a recurring line of states might be described by a fractional-linear map. If that map preserves three points, it must be the identity, while a decreasing one can only reverse their order. We would reject proposed criterions if an exact counterexample passed all local checks but generated infinitely many distinct predictions, or if incompatible cycle maps failed to give a valid obstruction. If the criterion is an immediate consequence of existing work, we would drop the claim and look for a sharper certificate or complexity bound only if a genuine gap remains.

After the first month one should have been able to workout some toy examples and observed if the phenomena suggests that there are reasons to belief an algorithm can be designed. The assumption of rank two matrix continuation might be too strong. The results might be very particular to this class of transducers and does not generalize nicely. In such case, we should look for other types of constraints that better capture the idea of studying explosion of states without computing the belief construction enumeratively.

Do trained LLMs have world models? Among the components of the systems that make an LLM agent (context window, weights, residual streams, output tokens…), can we describe them as Markov decision processes ?

LLMs are the essential of the frontier AI models at the moment, and are the technology behind the agents that have been involved in all the reported incidents so far. As World-model is a powerful framework to interpret the behaviour of an agent in terms of goal, interest, and strategy, being able to describe the internal state-beliefs and policies of an LLM could help understand their “reasoning” at a deeper level than what we can do with logit lens for example (we could maybe measure quantitatively their expected rewards via deception vs honesty).

We know how to probe the residual streams to see if some particular concepts are linearly encoded in the internal representations, which is a good indication of the richness of the information processed in LLMs. But this approach has two important limitations. First, it tells us if a prior human concept corresponds or not to any direction in the latent space, but cannot capture possibly new concepts that the model would use as an inner representation of the world. Secondly, this tool can be applied for a single input at a time as it depends on the residual stream, so it lacks generality. We can try to establish a mapping directly between the weights of the LLM and the state/action space and policy of its equivalent MDP description, and use Deepseek R1 as an experimental set-up for testing our theory.

Hypothesis: each token produced by the LLM is an action that has an effect on the environment, that can be, depending on the training/production phase, either a human user that provides new prompts as the conversation goes on, or an automatic program or human that provides a reward. If we successfully design the mapping, we could compute for a trained model :

• •the policy from the weights,
• •for each history (context window/chain of thought), the state-beliefs of the model

Remark: an MDP not only has a world model, but it is also able to perceive rewards and to update its policy to maximise its expected reward on the long term. Once the training is over, there are no rewards anymore per se, but if each state has a value, the observations given to the model via the prompts can maybe play an analogous role regarding the action state distribution by updating the state-belief?

The first step of the research would be to discuss options as to how we operate the mapping. Can we do it exactly or do we have to accept an approximation? Do we want to limit the number of states? Can we model some or all of the internal layers as a mental rollout?

Then, we can run some experiments:

• •What is the average error between the base LLM and its mapped MDP measured on
• •Can we control predictably and with precision how the outputs change when we tamper with the weights?
• •Can we design training datasets that poison/cure the model behaviour

A success for the first step would be to have at least one description of the LLM in terms of MDP, and especially a consistent definition of the state space.

My proposal is summarised in ‘Human summary’ at https://docs.google.com/document/d/1VzKrDJ55SCkdHrjHHS7nDLGhP_b8EM-orZg8ylq1j1k/edit?tab=t.0 · In a state-space model (SSM) trained on a cyclic task, the hidden-state transport carries a gauge-invariant, quantized loop holonomy that is not decodable by any linear probe of a single activation.

If the holonomy is found then a network can store learned structure that a local probe misses, because it lives in transport around loops, not at any single activation. The gauge-invariant loop-holonomy readout then becomes a new instrument for recovering content that current tools cannot see, and a demonstration that higher differential geometry has empirical relevance to representation learning. If instead the holonomy is zero or incoherent, then the linear representation hypothesis might be enough, and interpretability need not chase global structure. Either answer bounds what reading a model's representations can reach.

Some of the ingredients discussed exist in the literature, but I don’t believe the specific proposal has been considered. Closest is Sevetlidis & Pavlidis [1], who introduce gauge-invariant representation holonomy — activations as a field with a discrete connection, transported around input loops. But their estimator is built to vanish as the loop shrinks (O(r)) on contractible loops: it measures curvature, explicitly "not topological monodromy," leaving the flat, path dependent regime untouched. Javidnia [2] extends this to a gauge/sheaf atlas of superposition in LLMs, computing cycle holonomy — no flat/probe-invisible separation. Engels et al. [3] refute the one-dimensional linear representation hypothesis with circular features that are the unit for modular tasks — but a circle is locally decodable: curvature, not flat holonomy. Grokking [4] shows group representations computed as accumulated rotations, yet the state stays locally recoverable and no loop invariant is extracted. Sung [5] builds a network whose state is a non-abelian holonomy, but by construction, with no measurement protocol. Park et al. [6] formalize the linear hypothesis but stay pointwise. State-space models [7] make the connection literal — architecture, not evidence of stored, locally-invisible structure. None separates a flat, probe-undecodable holonomy from locally-visible curvature in a trained model.

I want to be honest that this is exploratory: I do not yet have a well-defined plan, and what I am proposing is to investigate these higher-categorical structures theoretically and empirically in toy models, using each to discipline the other.

Theory. Pin down the possible differential topology invariants in transformer type LLMS. I expect the reward interpretation (γ(t) as a flat 0-gerbe) to be the cleanest place to start, since there the geometry is forced by axioms.

Toy experiments. In parallel, build small SSMs on engineered cyclic tasks where I can put a candidate class in by hand, so the ground truth is known. The point is to see whether the theoretical objects are measurable at all, and whether a genuinely flat (probe-invisible) component can be distinguished from ordinary curvature.

The honest near-term goal is not a result but a sharpened, well-posed version of the question, plus a working measurement tool — the prerequisites for deciding whether any rung of the tower is real in trained models.

Prediction. At this exploratory stage my predictions are about feasibility. On the theory side, I expect the tower to be well-defined for a state-space model at the lowest rungs — connection and curvature, with γ(t) as a flat 0-gerbe the cleanest anchor — and to grow unclear, or vacuous, at the higher (gerbe / discrete-torsion) rungs. Empirically, in a toy SSM where I plant a class by hand, I expect a loop-holonomy readout to recover it as a gauge-invariant, quantized quantity stable across seeds, and at best to separate a flat (probe-invisible) part from ordinary curvature.

What would make me drop it. If the higher objects are not well-defined, or collapse to something already captured pointwise, the tower adds nothing. If the readout cannot recover a hand-planted class — near zero, or incoherent (failing quantization, gauge-invariance, drifting with tuning) — there is nothing to build on.

After the first month. A sharper question and the beginnings of the tools: a first pass at the theory dictionary for a state-space model (which rungs are well-defined, where a forgotten symmetry first registers), and a toy SSM with a hand-built class plus a prototype loop-holonomy readout — enough to see whether the objects are measurable at all.

What I would need. A small codebase and one GPU; more importantly, sanity checks from geometry on the higher-categorical claims and from interpretability on the setup.

If it became three years. First cut the real-model hunt; then the higher rungs (gerbes).