Proposals

When the combined dimension of separate beliefs about independent hidden processes far exceeds a transformer's residual width, can it still predict accurately, and how does it represent those beliefs within the available space?

A belief state describes what a model could infer about hidden causes from observations. Its geometry gives us a way to understand the model's internal representation. Recent work shows that transformers can store beliefs about independent processes separately, saving space. But what happens when even these separate beliefs need many more coordinates than the residual stream provides? This is the regime we expect realistic LLMs to operate in.

If accurate prediction survives, we should ask what information compression preserves and whether ordinary linear probes miss it. If prediction deteriorates, we can study which distinctions disappear first and how much width is needed for a chosen accuracy. Either outcome would help us interpret models of complex environments and assess when an incomplete picture of their beliefs reflects lost information or inadequate measurement.

Shai et al. [1] showed that transformers trained on hidden Markov models represent beliefs about hidden states. These beliefs describe probabilities assigned to possible states after observing a sequence. Their examples have enough residual dimensions to hold the belief geometry.

The 2026 follow-up [2] showed that transformers can represent independent processes in separate subspaces. This avoids storing a much larger joint belief. However, the separate beliefs together require ten dimensions, while the tested transformers have widths from 48 to 480. Thus, even the smallest model has room for them. My question concerns the next step: what happens when the factored beliefs themselves no longer fit?

Related work offers possible answers. Compressed predictive states [3] use random projections to reduce representation size. A random projection compresses a belief state into fewer coordinates using fixed, randomly chosen weighted sums of its original coordinates. This provides a simple baseline for testing whether transformers learn a better way to compress beliefs. Predictive feature theory [4] studies the resources needed for a given prediction accuracy. Superposition [5] shows how neural networks can pack features into shared dimensions. These results motivate the project but do not yet establish what a transformer learns in this setting.

The Johnson–Lindenstrauss lemma says random projections can approximately preserve distances between finitely many points. However, it does not guarantee accurate predictions or reliable belief updates. Comparing these different demands, including performance on previously unseen observation sequences, is part of the proposed investigation.

The project has two parts. First, I will construct compressed belief representations and establish what they can preserve. Second, I will test how transformers represent the same beliefs, using the first part as a benchmark for interpreting what they learn.

Theoretical construction

First, I will build a sequence generator from several independent Mess3 processes and calculate their exact beliefs after each observation. Each process has three hidden states, so its belief can be described by two probabilities. Observations will identify which process generated them; only that process advances. This keeps the beliefs independent and makes their combined dimension controllable. I will check that these coordinates vary independently and matter for prediction.

Next, I will study how well those beliefs can be compressed into fewer coordinates. I will compare random Johnson–Lindenstrauss projections, principal component analysis, and a network trained to preserve predictions. For each, I will measure prediction error and changes in distances between beliefs. This separates two meanings of good compression: keeping the original shape and keeping information useful for prediction. I will also examine what the lemma guarantees at the tested precision. While these compressions of the belief geometry may not be truly (or uniquely) optimal they will provide benchmarks for the transformer.

Transformer experiments

I will then train transformers on the same sequences, varying residual width above, below, and far below the combined belief dimension. I will compare their predictions with exact Bayesian predictions and the compression benchmarks, using the same observation history and comparable prediction heads.

Finally, I will test whether factor beliefs remain recoverable using linear and nonlinear probes. I will look for overlapping factor representations and test sensitivity to noise. Comparisons across layers and positions will check whether attention stores or retrieves information outside the final residual vector. Repeated training runs will establish which findings are reliable.

I expect transformers to retain predictions after the separate beliefs stop fitting into distinct residual directions, but with increasing errors as compression becomes stronger. I expect them to preserve distinctions that matter most for prediction, rather than all distances equally. Their representations may therefore predict better than random projections while distorting belief geometry more. Nonlinear probes may recover information that linear probes miss.

I would reject this prediction if random projections perform equally well across widths and repeated runs. Failure to recover beliefs with a linear probe alone would not necessarily establish that the belief geometry has been lost. I would also revise the interpretation if other positions or layers account for the information missing from one residual vector.

In three months, I would produce one working benchmark, comparisons with simple compression methods, transformer experiments across several widths, and a report with reproducible code. The first month would establish the generator and exact beliefs and develop compression benchmarks; the second would train transformers and begin their analysis; the third would continue to analyze their representations in a variety of residual widths. I would need GPU access for small models, existing hidden Markov model (HMM) code, and guidance on belief geometry and predictive compression. If time is short, I would reduce the number of HHMs and widths tested. If the project progresses faster than expected, I would explore compression in quantum or generalized predictive-state geometries, extending beyond the classical HMM belief simplices studied initially.

A finite-memory classifier sorts continuous inputs into three labels, guessing near boundaries and learning its prior from its decisions. Can this predictive system detect from its records that guesses were wrong, and when do bits describing uncertainty beat finer resolution?

A system that learns from its own decisions treats them as data. A wrong guess enters its prior, and the prior shapes later guesses. Whether these errors average out or build up decides whether its model of its inputs stays accurate. The same problem arises when models train on their own pseudo-labels or outputs [1, 2].

If the classifier can tell from its own memory that guesses have corrupted its prior, a system can know its model is failing without knowing which guesses were wrong. If memory is better spent describing how uncertain its prior is than on finer resolution, a design that expects to guess wrong beats one that only guesses less. Otherwise only more resolution helps.

Self-training on pseudo-labels can reinforce early mistakes [1], and models trained on their own outputs can drift from the true distribution [2]. In large learned models, error sources are hard to isolate. In my setting it is known: finite memory / resolution forces guesses near bin boundaries.

No device can always decide in bounded time which side of a boundary a nearby point lies on [3, 4]. My classifier measures two coordinates separately, so its cells are rectangles. With only two labels to decide upon, a resolution limitation is immediately apparent. With three labels arranged around a point, the boundaries meet at a junction that redrawing cannot remove (compare Chichilnisky [5]). Three boundaries cannot in general all lie along two coordinates, so some guesses cannot be designed away.

Computational mechanics asks what a predictor must remember about the past to predict a process [6]. The classifier's prior is such a memory, limited to p bits and built partly from guesses. I have not found work asking when such a self-built model stays accurate, whether the system can tell when it is not, or how to split a fixed memory between resolution, the prior and describing uncertainty.

Setup: Particles from a fixed source are points x in the plane, drawn around an unknown centre c near the junction of three arbitrary-shaped label bins. Decisions change only the classifier's memory of M = m + p + b bits:

• •Measurement (m bits): each bit splits the rectangular cell along one coordinate. If the final cell lies in one bin, the particle is sorted; otherwise it cycles, guessing with probability ε per step.
• •Prior (p bits): an estimate μ of c. Each decision moves μ a fixed fraction 2⁻ᵏ towards where the particle was placed: the cell centre if sorted, else the centre of the guessed bin's part of the cell.
• •Uncertainty record (b bits): a second estimate of c from sorted particles only. Its gap from μ signals guesses pulling the prior.

Guesses use μ to score bins. The most-probable rule picks the top bin, so guesses pull μ towards it and errors can accumulate; the sampling rule picks in proportion, so errors can cancel.

Step 1. Simulate both rules, measuring error rate, μ's distance from c and steps per particle, across m, p and source spread.

Step 2. Find when sorted particles keep μ near c against the pull of guesses, and the p below which rounding freezes μ.

Step 3. Test whether the gap grows exactly when the error rate grows.

Step 4. At fixed M, compare b extra measurement bits with a b-bit record that pauses guess updates when the gap is large.

Under the sampling rule I expect μ to stay near c. Under the most-probable rule I expect μ to drift towards one bin when guesses are a large share of all decisions, which happens when m is small or the source is concentrated near the junction, and the error rate to rise with it. I expect the gap to track this drift, and the record to beat extra measurement once m is large enough that one more bit removes few guesses.

Results that would change my plan:

• •The most-probable rule does not drift: errors do not accumulate here, and the question reduces to resolution.
• •The gap and the error rate do not rise together: the classifier cannot detect the problem from its own records.
• •Extra measurement always beats the record: describing uncertainty is not worth the bits.

Month 1: the simulation in Step 1.

Month 2: the analysis in Step 2 and the detection test in Step 3.

Month 3: the comparison in Step 4 and the write-up. Everything runs on a laptop.

If time is short, I will limit Step 2 to the simplest source. Steps 1 and 4 are the core results.

Unlearning a sub-process from a transformer’s training distribution withholds and mutates that knowledge rather than destroying it: the sub-process’s belief-state geometry is preserved or re-encoded, and how much survives predicts how easily it returns under relearning or in-context attacks.

Current unlearning evals argue for safety through behavioral similarity to baselines, often a base model that never saw the unlearning target and a known-harmful model that learned it. This relies on the elicitation power of the evals and leaves a gap: unlearned models look like base models until subjected to relearning, fine-tuning, or a change of framing. Belief state geometry provides an intrinsic baseline: the belief states an optimal predictor of the retained data must carry are fixed by the data-generating process. If the claim holds, retained knowledge can be read off the geometry directly. If it holds only for a nonlinear decoder, the knowledge has been re-encoded, not removed, and linear-probe audits can be defeated by unlearning itself. If a method instead collapses the forgotten geometry to that baseline, unlearning can erase a representation rather than hide it.

Shai et al. [1] showed that transformers trained on hidden Markov data linearly represent the belief-state geometry of the mixed-state presentation, and [2] extends this to factored processes; nobody has used that geometry as a reference for unlearning, where the target is the retained process’s own geometry. On the unlearning side, representation-level evaluations exist but all measure against an extrinsic baseline. [3] finds gradient ascent, NPO, RMU and IDK leave probes at ceiling while shifting outputs by up to 62 nats; [4] compares RMU, AltPO and others and finds a reversible regime where behavior is suppressed but geometry recoverable; [5] shows relearning recovers dominant components first; [6] found a “forgotten” model still probe-readable and [7] argues RMU is largely shallow. Each measures against a label or a trained model, so none can say whether a representation moved, re-encoded or disappeared, nor separate suppressing recognition of the forgotten data from suppressing the data itself. The mixed-state presentation gives both: a full geometry with a known metric, and a belief state that factors into which process is active and what it is doing. [8] plants a latent variable in natural text and recovers its belief geometry, the bridge to real models.

The early stage is a toy model where the geometry is exact. Each sequence comes from one of two hidden Markov processes, A or B, with Mess3 variants as the main pair; the mixture’s belief state is then a posterior on which process is active plus each process’s filter. I train a small transformer on the mixture, regress residual-stream activations onto the true belief states, and take decodability in excess of the same regression on a model trained on A alone. I then apply unlearning methods spanning output-level (gradient ascent, NPO), representation-level (RMU) and preference- or refusal-trained (AltPO, IDK) objectives, matched on forgetting and retention against the true predictive distributions, and record per layer the excess decodability of B’s geometry, its best rigid alignment before and after, how much of the change falls on the posterior versus B’s filter, and a nonlinear decoder as a second readout. Where geometry survives I steer with the fitted map to test gradient-free recovery, test in-context recovery with prefixes that force the posterior onto B, then relearn on a small B sample and count steps.

The later stage scales three ways: more components, making collateral forgetting measurable; harder structure, a switching process that moves between A’s and B’s regimes within a sequence, then product processes where A and B share states; and scale, a planted process in natural text unlearned from a small pretrained LLM, then an open-weight LLM with a synthetic forget set. An objective that targets A’s filter geometry directly is the stretch goal.

B’s belief-state geometry should survive unlearning above the retrain control, largely intact: rigid motion of the belief subspace or relocation to another layer, still linearly decodable wherever it lands. The change should fall on the posterior over which sub-process is active, or on the readout to the logits, not on B’s filter, so a prefix that forces the posterior onto B should restore B behavior before any gradient step. Re-encoding that only a nonlinear decoder recovers would surprise me. In the switching process, unlearning should act on the transitions into B’s regime; in product processes, survival should fall with how entangled the emissions are.

I would drop the question if B’s geometry is not decodable above the control before unlearning, or if recovery under attack does not track surviving geometry. A method that drives both decoders to the control level would falsify the claim and be the first demonstrated erasure.

Three months would produce a definition of unlearning success stated against the data-generating process rather than a reference model, and a taxonomy of methods by what they do to belief-state geometry, with controls, layer profiles, the posterior/filter split and the three-attack ladder. Month one is the mixture, which is the floor; month two the switching process and more components; month three product processes and the planted-process LLM bridge. The sweeps need one GPU for several days and the epsilon-transformers tooling. I would drop the geometry-targeting objective first, then the open-weight LLM, then product processes.

Are intermediate-layer belief state geometries interpretable?

Understanding how transformers construct and transform internal representations supports a more rigorous approach to alignment. An intuitive account of how concepts develop across layers could clarify the effects of steering and ablation, the interpretation of linear probes, and the mechanisms underlying generalization.

Studying these questions in a controlled toy model lets us compare representations against known predictive structure, helping distinguish incomplete computation from uncertainty in the data and identify hypotheses to test in larger models.

Prior work examines layerwise belief-state geometry in simple systems where belief states can be encoded almost entirely in the first layer. These experiments also span multiple context lengths, including contexts that leave uncertainty about the hidden state, making it difficult to distinguish this uncertainty from incomplete computation in intermediate layers.

By fixing the context length so that every valid prompt uniquely determines the hidden state, we isolate how representations develop across layers toward the final belief-state geometry.

We hypothesize that belief-state construction in intermediate layers is limited by the capacity of the attention block to facilitate interactions between pairwise tokens. Using this consideration, we can predict which belief states are constructed at each layer. Preliminary results on a simple system (Z1R) currently match this picture.

We would like to study this result through the following steps:

Derive the per-layer belief geometry in simplified models from restricted-context considerations.

Verify the results across several datasets and architectures.

Connect this framework to relevant safety tools, such as ablation and steering, by developing a theoretical account of steering and ablation in simplified HMMs and validating its predictions against trained models.

Understand the conditions under which this picture fails, and whether the general idea is applicable to LLMs.

We can make exact predictions about which belief states should appear in each layer for a given model with a fixed context size. Large context sizes should typically result in belief states at the corners of the simplex in the final layer. Indeterminate belief states should be present predominantly in early layers and vanish as more “effective context” is provided through belief-state interactions.

Failure of our predictions would change our mind, but even a negative result would be interesting.

In three months, we could reasonably construct a theoretical formalism to describe the emergence of belief states through pairwise interactions of prior states via the attention block at each layer. From this, we should be able to offer insights into common techniques relevant to AI safety, such as ablation and steering, within this formalism.

We believe there is a clear connection between intermediate belief states in these models and the formation of concepts and features in LLMs.

Question: How do multi-scale relations in HMMs reflect on the belief state geometry of transformers? · Hidden Markov Models (HMMs) can exhibit multi-scale structure and can be coarse-grained, which will reflect on the belief state geometry of transformers.

For humans, “what makes something a whole” is a basic question. The first time I see a chair, I might see four legs and a piece of wood, after enough times it becomes its own thing. If I saw an alligator on a desk a million times, I would eventually make it a single object, a “Deski-gator”.

Transformers may do the same, concepts that often appear together and whose combination predicts something their parts do not may become their own feature. If so, a combination of multi-scale structures and computational mechanics could explain how. If not, either a different perspective is needed or transformers do not work this way. Either answer directs future research.

There are existing works on looking at hierarchical structure, from computational mechanics [2], HMMs [1, 3], grammars [4]. The most important gap is how this theory appears in transformers.

[1, 3] explores “lumpability” in HMMs, and gives theoretical ground to possible hypothesis. It does not do any work on the geometry, or empirically verifying the theory on any trained model.

• [2]is the most relevant, creates a computational mechanics framework for looking at the coarse graining. It seems these hierarchies are not validated on trained transformers and their belief state geometry, where my question remains open.
• [4]shows deep networks learn synthetic hierarchical grammars, but does not touch on the computational mechanics or belief geometry.
• [5]Explains the factored representations in transformers, also very relevant. Multi-scale approach is parallel to this paper: instead of decomposing the HMMs into separate HMMs, we find smaller HMMs within. The multi-scale tools are needed to get a full picture of what is going on in transformers.

A combination of the ideas can explain about the emergence of features and why these features are (not) “simple”, from the view of hierarchical information. This direction is not studied as well, to the best of our knowledge.

The first concrete step is to construct concrete HMMs with an obvious coarse-graining. One example is the Mess3, but each state can be expanded to be 10 micro-states with high probability of remaining within the cluster, with the micro and coarse-grained states emitting different tokens to distinguishing them. The transformers can be trained on the outputs of these models, and see if there is a relationship between the belief state geometries.

Important questions to be answered:

30 State HMM Geometry: It is hard to visualize the geometry of a HMM with high states. So a theoretically grounded way of comparing the geometries is required.

Geometry Comparison: How could these belief state geometries be compared? Theoretical results exist in [1], this could be used to hypothesize on what structure to expect in the geometry. Simple visualization can also be enough, (e.g. PCA).

If successful, more ambitious goals can be done for languages. For a known hierarchical structure in language, we can look for relations in their belief states. Comparative metrics can be developed to avoid exactly constructing the belief state geometry.

Final goal is to connect to linear representation hypothesis (LRH). A "linear direction" is the simplest geometric object. Can be just described by a 1-dimensional "direction" vector. We can connect the empirical evidence for LRH with the hierarchical approach. The “minimal” elements of the hierarchy might be the “simplest”: beliefs about a binary coarse variable lie on a line segment, which is exactly one linear direction possibly explaining LRH.

I would expect to see:

-Belief state geometries of the original and the coarse-grained HMM are related, especially when coarse-graining is exact.

-Data statistics cause models to combine micro-concepts to create macro-concepts: the combination must predict something its parts do not, and appear often enough (e.g. first time we see a chair, it is not its own concept -- after 1000 times, it is)

-“Concepts” with “simpler” belief geometry corresponds to the examples we see in the linear representation hypothesis. This can possibly be explained via data statistics, and the hierarchical view.

If these observations hold, could give a nice theoretical framework to decompose features (coarse-grain if can, and factor as in [5] if can). If exact and broken coarse-grainings look the same in the transformer, I would change my mind about how hierarchical reasoning and combining features happen in transformers.

Three months could produce:

Concrete theory: What lumping of HMMs (coarse-graining the tokens) corresponds to in the belief geometry.

Toy experiment 1: In hand-crafted HMMs, whether this geometric relations can be observed

Toy experiment 2: When does a transformer decide to “combine” concepts into a whole, and when not -- with a hand crafted HMM and this multi-scale approach.

Extension to language: Specifically selected samples from language, can also demonstrate this.

If more than three months, I would drop the final point. Working on toy examples and a solid theory could be the groundwork for future empirical research in LLMs.

In transformers trained on HMMs, the first layers tend to learn a constrained belief geometry, which is later refined towards the full belief geometry. Which layer-layer interactions sustain such an information process? Can this be shown rigorously in simple models?

Understanding this would give us knowledge about how the belief state geometry is learned in a transformer. If constrained beliefs arise because early stages preserve mainly single-source evidence while later interactions integrate information required for the full posterior, we obtain a mechanistic explanation of why depth is useful. It would also give principled information-theoretic quantities for locating where such integration occurs and possibly allowing safety interventions. If this fails, that is also informative: constrained and full belief geometries may be descriptive projections rather than successive computational stages, or the relevant integration may already occur within a single attention block.

Piotrowski et al. [1] derive constrained belief updates that explain geometric structures in transformer attention outputs and investigate how subsequent processing changes those structures. In particular, they empirically observe that initially a constrained belief is learned, which is then refined towards the full belief. Shai et al. [2] establishes a connection between transformer representations and the belief geometry of hidden-state processes, including information relevant to longer-term prediction. These results provide concrete targets for understanding representations, but leave open a general account of which interactions compute the corrections separating constrained from full beliefs.

The authors (one of them is part of this proposal team) of [3] investigate relationships between higher-order mechanisms and synergistic information, while [4] use information decomposition to characterize distributed information during learning. These approaches can distinguish information available from individual sources from information accessible only through their joint observation. While these results highlight potential connection between the information properties of activations and the underlying mechanisms, they do not characterize systematically the integration of information in a controlled context with known learning mechanisms as the ones of the transformers.

We would first compute exact and constrained beliefs for families of finite-state HMMs, including simple cases where correct prediction requires combining information from multiple observations. For hidden state $S_t$ and observations $X_{1}$, let

denote the full Bayesian belief and $c_t$ the constrained belief of [1]. We would characterize what predictive information is missing from $c_t$, and under what conditions recovering $b_t$ requires genuinely joint processing of several observations.

We would first study this in analytically tractable models, such as simple linear or shallow nonlinear networks, and ask when an additive or constrained computation naturally precedes a more integrated computation. We would then investigate which parts of this picture generalize to attention.

Next, we would train small transformers on these HMMs and measure constrained-belief recovery, full-belief recovery, and future prediction after successive attention and MLP stages. For fixed source groups, such as messages associated with different evidence positions, we would use target-specific information decomposition,

to study when predictive information becomes available only through joint processing of multiple sources.

Finally, we would intervene on candidate interactions while preserving single-source responses as closely as possible, testing whether this selectively disrupts the refinement from constrained to full beliefs. Mutual-information and activation-based controls would assess whether the information decomposition adds explanatory value.

• •Earlier computational stages should be better described by constrained beliefs, while later interactions should improve recovery of the full belief.
• •HMMs requiring stronger integration across observations should show a larger constrained-to-full refinement and a greater dependence on depth.
• •Information-decomposition measures should identify where predictive information becomes jointly accessible.
• •Intervening on these interactions should selectively damage full-belief recovery while preserving constrained-belief information.

We would change our mind if full beliefs are already recoverable at the earliest stages, or if the information measures do not track the refinement well, analyzing the failure modes would then be the focus.

• •Study analytically tractable HMM families and one simple neural model where the constrained-to-full transition can be analyzed theoretically.
• •Train small transformers and measure constrained/full belief geometry across attention and MLP stages.
• •Test the proposed information-integration mechanism using specific safety interventions e.g. steering.
• •Use saved training checkpoints as a secondary experiment if time permits.
• •If the project becomes too large, we would drop broad HMM comparisons and related ideas on training dynamics, continue to focus on the theoretical toy model or depth-wise analysis to extend the results of [1].

When generative models are trained only to learn distributions, what predictive information their internal representations contain? Can we distinguish representations that are necessary for distribution learning from those that emerge because of architecture/optimization, across autoregressive, diffusion, and flow-based models?

Generative models are optimized to learn probability distributions, not to identify latent variables, learn predictive states, or construct interpretable representations. Nevertheless, their hidden activations often exhibit structured information about the data generating process.

Why should this happen? Is learning a useful representation an unexpected consequence of accurate distribution learning, or merely one convenient solution found by a particular architecture?

Answering this would connect generative modeling, statistical sufficiency, dynamical systems, and mechanistic interpretability. It could clarify what internal structures are genuinely required for successful prediction, which structures are artifacts of the training procedure, and whether different generative paradigms discover equivalent predictive information.

It would also provide a principled foundation for evaluating and eventually improving the reliability and generalization of learned representations.

Computational mechanics [1] establishes that causal states provide minimal sufficient representations for predicting stochastic processes. Shai et al. [2] demonstrate that autoregressive transformers trained on hidden Markov processes can linearly represent Bayesian belief-state geometry, including nontrivial fractal structures. Piotrowski et al. [3] further explain how attention can implement constrained Bayesian belief updating. These are important results, but they do not establish that a certain internal geometry is necessary for distribution learning.

Other generative objectives introduce a different problem. Masked diffusion language models [4] learn denoising conditionals, while flow-based language models [5] learn posterior predictions associated with noisy continuous interpolants. Both involve conditional inference, but their information states differ from those of left-to-right prediction.

Peng et al. [6] show that explicitly aligning autoregressive and diffusion representations can accelerate training. That does not determine whether alignment arises without representation supervision. Tang et al. [7] demonstrate why denoising quality must also be distinguished from sampling correctness.

The central unresolved issue is therefore not simply whether belief states can be decoded. It is about which predictive information is induced by the generative objective, how it is represented, and whether the model uses it meaningfully.

Even perfect distribution learning does not uniquely identify hidden coordinates. Moreover, an autoregressive transformer can revisit its history, whereas a recursively updated state-only generator must preserve information needed for future predictions. This distinction provides a concrete starting point.

First, establish a ground truth. Construct finite-state stochastic processes with analytically computable predictive distributions: an i.i.d. and a first-order Markov control, the Even Process (from Paul's lecture), and a hidden Markov model containing distinct histories with identical next-token distributions but different longer-horizon predictions. Extend to Mess3 or RRXOR to study more complicated belief geometry.

For each process, compute the Bayesian filtering beliefs, predictive-state equivalences, future-block distributions, and corruption-conditioned smoothing posteriors.

Second, establish what distribution learning mathematically requires. Relate the excess predictive log-loss of a representation to conditional mutual information. Prove that a predictive model whose entire future state update and output depend only on a recursively updated representation must preserve sufficient predictive information. Explain why this implication need not hold for an individual transformer activation when other parts of the history remain accessible.

For denoising models, derive the posterior targets using forward-backward inference. Keep filtering, smoothing, and full-sequence generation separate.

Third, train and compare models. Train small recurrent autoregressive models, GPT-2-style causal transformers, and masked-diffusion transformers on identical synthetic distributions, without any representation supervision. Use matched architecture capacity and reported training compute where possible. Extend to continuous categorical flow matching once the first comparison works.

Measure three distinct outcomes: distributional fidelity against the oracle; predictive information recoverable through linear and nonlinear probes, including longer-horizon predictions; and causal relevance through controlled activation interventions. Compare belief geometry and predictive-distribution geometry, and test rare histories and lengths beyond training.

Separate denoiser errors from sampling errors. In particular, assess whether a well-predicted posterior actually leads to a correct generated joint distribution.

Fourth, extend to realistic sequence modeling. Transfer the diagnostics to nanochat/GPT-2-style training and flow-based language models. Investigate whether predictive representations identified in controlled processes remain useful for understanding longer-context language modeling.

What I expect to see: A sufficiently accurate autoregressive generator with a state-only update must retain enough information to distinguish histories that agree on the next-token distribution but differ in their future distributions. A full-context transformer may distribute that information across layers or cached history rather than a single activation.

Diffusion and flow-based denoisers should approximate oracle corruption-conditioned posteriors when well trained, potentially giving rise to smoothing related representations rather than the filtering representations most natural for autoregression. The resulting internal geometries need not agree even when the learned observable distributions do.

Some representations may be linearly recoverable but causally incidental. Others may be necessary while only nonlinearly accessible.

What would change my mind: If the observed geometries disappear under seed, capacity, and probe controls, or cannot be causally connected to predictions, I would not call them essential. If differences between model families disappear after matching their conditioning information and performance, I would reject a claim of objective-specific representation geometry.

A negative result would still clarify what distribution learning does, and does not, identify.

Two weeks: Derive the predictive-sufficiency benchmarks and the next-token-aliasing example. Implement HMM filtering and smoothing, train small autoregressive and masked-diffusion models, and conduct an initial probing and intervention study. The decision point is whether the models approach oracle predictive performance closely enough to make representation comparisons meaningful.

Three months, conditional on the pilot: Add categorical flow matching, larger belief-state geometries, multiple seeds and context lengths, and systematic causal tests. Develop the theory and produce a reproducible benchmark and short paper. Consider GPT-2/nanochat-scale experiments after the controlled results are established.

Requires small neural models, standard GPU access, and dynamic programming oracles. Large-scale language modeling is not necessary for the initial contribution.

Do transformers appear to learn marginal beliefs first because they prefer factorization, or because interaction information is less predictive or harder to extract? Can changing only the input encoding reverse that order?

Computational mechanics characterizes predictive state structure; we ask how that structure emerges during transformer training, and whether marginal beliefs are learned before joint beliefs.[4] Accurate marginal predictions early in training can coexist with errors about interactions, which matters for alignment when safe decisions depend on feature combinations that individual-feature probes may miss. We will match how much marginal and interaction information reduce loss, then reverse their relative computational difficulty to test how input encoding affects learning order.

• •Shai et al. observe low-dimensional representations early in training, followed by expansion in a setting where noise breaks conditional independence. [1] Their dependent-chain setting stays factored. These findings raise the question of when models learn about individual factors and their interactions.
• •Some possible explanations are factorization bias, less predictive information about interactions, or more computation needed to extract it. [2][3] All these can overlap since a factorization bias could arise from easier optimization.
• •It is not clear whether marginal predictions would still improve first if interaction information were equally predictive and easier to extract. This comparison would help us understand how predictive value and input structure shape learning order.

• 1.Start with a toy data generator and a small transformer trained from scratch to predict binary outcomes A and B, encoded as −1 or +1. The pairwise moment E[AB] is the expected product given the observations, measuring agreement versus disagreement.
• 2.Provide separate signals about the marginals and the pairwise moment. Use joint Brier loss plus weighted marginal losses, comparing conditions where interaction information has lower or equal predictive value.
• 3.Construct two invertible input encodings: marginal prediction uses one observation source and pairwise moment prediction requires combining two, then swap. Preserve available information, target probabilities and sequence length; tokenize observations separately.
• 4.First train separate models on marginal prediction and pairwise moment prediction. Check that the encoding reverses which task is easier, then fix settings before joint training.
• 5.Compare accuracy on these tasks across several seeds, using the same starting weights and training time for each pair. Track representation dimensionality alongside prediction accuracy.
• 6.Repeat with weighted cross-entropy, matching predictive values again, to check whether the learning order depends on the loss.

We expect pairwise moment prediction to improve earlier as its predictive value increases, and learning order to reverse when computational difficulty is reversed at matched predictive value. A persistent marginal lead would suggest that the separate-task difficulty measurements do not fully explain joint training.

We will also test whether low-dimensional representations coincide with accurate marginals and an inaccurate pairwise moment. If the baseline shows no consistent marginal-first order across seeds, we would drop that framing and focus on how geometric changes relate to predictive accuracy. Repeating the comparison with cross-entropy will test whether the learning-order effect depends on the choice of loss.

Month 1: generator, loss calibration and separate-task difficulty checks

Month 2: paired joint-task runs, learning curves and activation dimensionality

Month 3: cross-entropy comparison, uncertainty and release

Deliverables: controlled tests of learning order and a comparison of geometry with marginal and pairwise moment accuracy. Follow up would be to test this mechanism in Shai et al.'s HMM setting

Can the correlators of a neural-network field theory, together with appropriate symmetry and consistency constraints, determine the scaling exponents governing feature emergence in grokking transitions? Under what conditions are these exponents uniquely identifiable?

Feature emergence and grokking provide tractable settings in which to study qualitative changes in learned representations. Neural-network field theories offer a statistical description of network outputs and their correlators, while conformal field theory provides tools for constraining operator spectra and scaling dimensions.

Connecting these frameworks could establish whether some properties of learned representations can be predicted from structural constraints, rather than inferred solely through numerical fitting. This would provide a theoretical foundation for understanding representation formation and potentially contribute to mechanistic interpretability.

The question is open because we do not currently have an established theory showing that conformal data from a neural-network field theory determine the scaling exponents of a learning transition such as grokking.

The initial objective is to reproduce the relevant analytical and numerical results of Rubin et al. for modular addition, identifying the specific observables that characterise feature emergence and the scaling laws, if any, that govern them. A concrete table of observables with relevant scaling laws is found in table 1 of [2].

For the initial project, I would define a field corresponding to the feature whose emergence enables generalisation. For example, in Modular Addition, I would define the following fourier mode:

For modular addition, let k denote a target Fourier feature and define its learned amplitude by

The learned amplitude of the k-th Fourier feature is defined as

A candidate feature observable is then

If an appropriate field operator Oₖ can be constructed in the NNCFT description, its two-point function would be

Here, x₁ and x₂ denote points in the domain on which the NNCFT is defined, rather than training times.

I would investigate whether these observables admit a description within the neural-network conformal field theory framework. This requires establishing a correspondence between the relevant neural-network ensemble, its parameter-space measure, the operators describing learned features and the correlators used to characterise them.

I predict that, under appropriate assumptions, correlator consistency conditions in a neural-network conformal field theory will constrain the scaling dimensions of operators associated with learned Fourier features in modular addition. However, I expect that these constraints alone will generally be insufficient to uniquely determine the dynamical exponents governing feature emergence during grokking, without additional information about the training dynamics.

I would revise this prediction if I found that the relevant correlator constraints, together with independently justified symmetry and consistency conditions, uniquely determine a measurable scaling exponent that agrees with numerical experiments across system sizes and training conditions, without using that exponent as an input. I would abandon the proposed CFT-based approach to predicting grokking exponents if I established that no consistent operator correspondence exists for the relevant learning observables, or that the available conformal data cannot constrain the target exponents even after incorporating the physically justified assumptions of the model.

Formulate a precise set of correlator consistency conditions and determine whether they uniquely constrain specified scaling exponents. If they do not, establish a no-go result under clearly stated assumptions. In order to do this, I would need to work primarily on adapting the framework of the NNCFT paper to the setting of the grokking phase transition.

Can the conservation laws of neural network gradient flow be used to develop a perturbative theory of learning dynamics? In particular, can statistical mechanics and Singular Learning Theory provide quantitative descriptions of learning and generalization conditioned on dynamical invariants?

The training dynamics of neural networks depend not only on the loss landscape, but also on the parameterization and initialization of the model. In particular, gradient flow may admit conserved quantities that constrain the evolution of the parameters, despite the dissipative nature of the dynamics. These constraints provide additional information about the trajectories accessible to the learning process, and may therefore be used to characterize different learning regimes and their generalization properties.

The purpose of this project is to investigate whether conservation laws can be systematically incorporated into the mathematical description of neural network training. More precisely, we aim to connect dynamical invariants with statistical mechanics and develop a perturbative framework applicable to increasingly general architectures.

Consider a deep linear network with weight matrices $W_l$, trained by gradient flow. The balancing matrices

are conserved along the dynamics [Du2018]. For positively homogeneous activations such as ReLU the matrix laws generally fail, but their diagonal components remain conserved. This suggests $G_l(t)=D_l+R_l(t)$, with $D_l=\operatorname{diag}(G_l(0))$ constant and $R_l(t)=\operatorname{offdiag}(G_l(t))$ not conserved in general. Whether, for small off-diagonal components, the dynamics admit a perturbative description in terms of the conserved quantities is open: conservation laws and implicit regularization have been studied [Du2018, Woodworth2020], but a general perturbative theory is missing.

A second difficulty is statistical. The invariants partition parameter space into sectors that gradient flow cannot cross. In conservative systems this motivates generalized Gibbs ensembles, where conserved quantities enter through Lagrange multipliers. Training is dissipative, so we regard the ensemble as an auxiliary description, not a stationary measure of the dynamics. The questions are which properties of gradient flow it recovers, and whether restricting to an invariant sector modifies the singular geometry governing generalization [Watanabe2013].

The long-term goal is a perturbative theory of nonlinear training: for small $R_l(0)$, control $R_l$ over finite time intervals and derive approximate conservation laws with error bounds (a perturbative problem for a dissipative system). As a first step we study a solvable model, the diagonal linear teacher–student regression

For gradient flow of a loss depending on $w=u\odot v$, the quantities $\Delta_i=u_i^2-v_i^2$ are conserved and govern the kernel-to-rich transition [Woodworth2020]. The behavior of gradient flow in this model is known; the new contribution is its statistical description, through Gibbs ensembles and singular geometry. Since the model is simple, we can study two limits.

SLT limit ($d$ fixed, $n\to\infty$).

We consider

with $K$ the population KL divergence, and ask whether the RLCT and multiplicity depend on $\delta$, and whether this gives invariant-dependent generalization predictions.

Thermodynamic limit ($d\to\infty$, $n=\alpha d$).

We consider the generalized Gibbs partition function

and its free energy and order parameters, compared with gradient flow, whose interpolating solution is characterized by an initialization-dependent potential [Woodworth2020]. We ask how the two descriptions connect as $\alpha\to\infty$.

We compare the descriptions on their prediction of the generalization error, and against numerical gradient-flow simulations.

We expect the conserved quantities to classify the dynamically accessible regions of parameter space, characterizing learning regimes through initialization-dependent constraints. In the SLT limit, restricting to invariant sectors may modify the singular geometry, giving invariant-dependent Bayesian generalization predictions. In the proportional regime, the generalized Gibbs ensemble may reproduce some statistical properties of gradient flow, while differing where dissipation matters; understanding these differences is itself an objective. More generally, these results may provide the first elements of a perturbative theory in which exactly conserved quantities determine a reference dynamics and non-conserved components are treated as corrections.

The short-term objective is a mathematical analysis of diagonal linear networks, not a perturbative theory for arbitrary architectures. Within three months, we aim to obtain analytical expressions or controlled approximations for the constrained and generalized partition functions, compare Gibbs predictions with gradient-flow simulations, and compute the RLCT on selected sectors. If the disorder average in the thermodynamic limit proves too heavy, we restrict to a Gaussian design with numerical checks. Even one example exhibiting a Gibbs–gradient-flow discrepancy or an invariant-dependent RLCT would be a meaningful result.

In tanh autoencoders squeezed below their information content, solutions that drop or coarsen bits have a lower learning coefficient λ than packing solutions, so networks switch to packing only above the critical data size predicted by nL + λ log n.

Cammarata argues that squeezed networks get coarser rather than packing, so their neurons stay readable; this is central to whether safety audits can trust a model’s own neurons.

If the prediction holds, we can say in advance, from λ and the amount of data, when a network will pack information into its neurons and when it will keep them clean. That tells auditors which networks and training regimes neuron-level explanations can be trusted in, and turns a qualitative debate into a quantitative one.

If it fails, the learning coefficient does not explain the choice between dropping, coarsening and packing, so some other factor, such as the training dynamics or the nonlinearity, decides it, and neuron audits need a different theory.

The closest work is Chen et al. [1], who applied singular learning theory to the toy model of superposition [2]. They showed that λ and the free energy predict which solution the Bayesian posterior prefers and when it switches, a Bayesian phase transition. This is the method we need. But that model is a linear bottleneck whose readout is monotone in each neuron, so it cannot pack dense bits into one neuron’s activation levels; dense features there are simply dropped [2]. Dropping, coarsening and packing have never been compared.

Cammarata [3] ran the squeeze test on a real vision model: cutting a layer from 480 neurons to 10 made its neurons coarser rather than packed. Without ground truth, no one can say what was lost, and nothing predicts when this would fail.

Our pilot, on Rorschach images whose bits are all known [4], showed that it does fail in some cases: ReLU networks dropped bits while tanh networks packed them. No existing theory explains when each happens.

We start by constructing and analysing a minimal model. Inputs are two random bits; targets are the same bits, each flipped with probability ε, so the true distribution is reachable with finite weights. One tanh neuron h = tanh(w·x + b) feeds a three-unit tanh decoder with two Bernoulli outputs (17 parameters). By counting the distinct values of h, its solutions are drop (2 values), coarsen (3, tracking x₁ + x₂) or pack (4). We compute their losses exactly: drop and coarsen both give H(ε) + log 2 per sample, pack gives 2H(ε). We also check which of the three are local minima under a bounded prior.

Next we compute λ for each solution by expanding the KL divergence around it and resolving it to normal-crossing form [5]. We check each value numerically by volume scaling and with the SGLD estimator [6].

These λ values and losses go into the free energy nL + λ log n. Since drop and coarsen tie on loss, λ alone ranks them; packing should win above a critical data size n*, which we compute as a function of ε.

We then test the prediction. We sample the posterior with SGLD and train many seeds over a grid of n and ε, label each run by its number of values, and compare the observed switch with n*, as Chen et al. did for superposition [1].

Finally we estimate λ in the Rorschach networks [4] below 19 neurons, to see whether the same ordering explains why tanh packs.

We predict that packing has the highest λ of the three solutions. The free energy then favours dropping or coarsening at small n and packing above n*, and we expect the observed switch in posterior samples and trained runs to fall within a factor of two of n*. We don’t know whether dropping or coarsening has the lower λ. If it is coarsening, singular learning theory backs Cammarata’s “coarser” picture at small data. In the Rorschach networks, we expect tanh solutions that pack to show the same λ ordering as the toy model.

We would drop the question if packing does not have a higher λ, or if the switch in trained networks shows no link to n*, for example if it is off by more than ten times or depends only on initialisation. Then λ does not decide the outcome, and training dynamics is the better frame.

Three months would produce:

Step 1: the minimal model’s losses and exact λ for all three solutions, checked numerically.

Step 2: the predicted n*(ε), tested against posterior samples and trained runs.

Step 3: λ estimates in the Rorschach networks, and a short paper with code.

Needs: one CPU for the toy model, about a GPU-week for SGLD on the larger networks, and a mentor who knows singular learning theory. If it runs long, we drop the Rorschach step first; the toy model alone answers the question.

Can Singular Learning Theory be recovered from the thermodynamic limit of high-dimensional statistical models? In particular, do the thermodynamic and low-temperature limits commute, and can the resulting free energy be described through emergent statistical properties of singular parameter-space geometries?

Singular Learning Theory (SLT) provides a mathematical framework for statistical inference in models with singular parameterizations, including neural networks. The asymptotic regime underlying this theory is, however, formulated at fixed parameter-space dimension.

To place this regime in the broader framework of statistical physics, consider a family of partition functions of the form

where $H_N$ is the Hamiltonian, $\pi_N$ is a reference measure on the parameter space, $N$ denotes the system size, and $\beta$ is the inverse temperature.

Two distinct asymptotic regimes are naturally associated with this expression: the low-temperature limit $\beta\to\infty$ at fixed $N$, and the thermodynamic limit $N\to\infty$ at fixed $\beta$. In the former, the Gibbs measure concentrates near the minima of the Hamiltonian. In the latter, the increasing number of degrees of freedom may give rise to collective phenomena and emergent order parameters.

Although finite systems may already exhibit collective behavior, the thermodynamic limit provides the natural framework for describing macroscopic order and genuine phase transitions. Such phenomena are not directly captured by fixed-dimensional low-temperature asymptotics.

Moreover, the two limits need not commute. Rather than representing a limitation, their non-commutativity may encode different physical properties of the system.

In Bayesian inference, the number of observations $n$ plays the role of an inverse temperature, since the likelihood enters the partition function through the factor $e^{-nL_{n,N}(\theta)}$, where $L_{n,N}$ is the empirical average negative log-likelihood.

From a geometric viewpoint, it is natural to consider instead the population Kullback–Leibler divergence $K_N(\theta)$ and the corresponding partition function

For fixed $N$, the asymptotic behavior of this integral can be studied by Laplace's method. The main difficulty in singular models is that the set of minima may form a possibly singular analytic variety, preventing a direct application of the classical non-degenerate Laplace approximation.

Using methods from algebraic geometry, Watanabe's theory provides, under suitable assumptions, the expansion

where $\lambda_N$ is the real log canonical threshold and $m_N$ is its multiplicity [Watanabe2013].

The expansion is obtained at fixed $N$, and its remainder is not necessarily uniform in the dimension of the parameter space. Consequently, it does not directly determine the thermodynamic limit or justify exchanging the two limiting procedures.

On the other hand, statistical-mechanical approaches to large neural networks already investigate thermodynamic limits and collective order parameters [Pacelli2023, Li2021]. The relation between these descriptions and the geometric invariants appearing in SLT remains to be understood in general.

The purpose of this project is to investigate whether a thermodynamic description of singular statistical models can be obtained in terms of distributions of suitable geometric observables, and whether the subsequent low-temperature limit allows one to recover, or extend, the asymptotic theory of Watanabe.

We propose to investigate the thermodynamic limit of the normalized free energy,

followed by its low-temperature asymptotics as $n\to\infty$.

Our research will proceed along three directions.

First, non-commutativity of limits.

We will construct families of singular statistical models with an increasing number of parameters and investigate whether the classical fixed-dimensional asymptotic expansion remains valid in the thermodynamic regime. This includes identifying suitable energy normalizations and comparing the two orders of limits after removing their leading divergences.

Second, random geometry.

We will investigate whether collective geometric observables, such as distributions of local degeneracies, Hessian ranks, or singularity types, admit well-defined thermodynamic limits. The goal is to understand whether the free energy can be related to the statistical distribution of these observables, rather than exclusively to fixed-dimensional singularity invariants, following the concentrations of order parameters picture in statistical physics.

Third, quantitative predictions.

Rather than fixing a model a priori, we will select it by explicit criteria: (i) exact or numerically controlled free energy at finite $N$; (ii) known or computable RLCT at each $N$, so that the fixed-dimensional prediction exists; (iii) a natural scaling of $K_N$ and $\pi_N$ with $N$ yielding a non-trivial limit; (iv) coupling between parameters, since factorizing models make the limits commute trivially; (v) measurable geometric observables. We will start from a non-interacting control satisfying (i)–(iii), then add coupling, and compare finite-size free energies against classical SLT and the thermodynamic description.

Importantly, we do not aim to prove a general theorem for arbitrary neural networks within three months. Even one explicit model exhibiting non-commutativity, or a demonstrable improvement in finite-size predictions, would constitute a meaningful first result.

Our main hypothesis is that the thermodynamic and low-temperature limits do not commute in certain classes of singular models. In these cases, the large-dimensional free energy may depend on collective geometric quantities that are not captured by the fixed-dimensional RLCT and its multiplicity alone.

We expect that a statistical description of singular geometries could provide more accurate predictions when both the model dimension and the number of observations are large, particularly outside the regime where classical asymptotic expansions are uniform.

A more ambitious possibility is that Watanabe's invariants can be recovered as low-temperature characteristics of an emergent random geometry.

However, commutativity would also be informative. If the limits commute in our examples, we will investigate the structural conditions responsible for this behavior and determine the regime in which standard SLT predictions remain reliable.

The immediate objective is a mathematical investigation of tractable singular models, rather than a complete theory of deep neural networks.

Within three months, we aim to produce at least one explicitly analyzable model, a rigorous or quantitatively controlled comparison of the two limiting procedures, and numerical results testing the accuracy of thermodynamic versus fixed-dimensional predictions.

If the random-geometric formulation proves tractable, we will also attempt to identify its relevant collective observables.

The longer-term goal is to extend this framework to neural networks, connecting singular learning theory with the statistical mechanics of large-scale learning systems.

Recent work by the Simplex has shown belief geometry emerging gradually [1, 2], with deeper details of the fractal appearing at different training times. How does the effective model complexity, as measured by the Local Learning Coefficient, change as the model learns richer detail of the belief-state geometry?

To know whether our models generalise beyond training, we want to know whether they build a representation of the world beyond the tokens.

Shai et al. showed that transformers trained on the tokens of a hidden Markov model build approximations to the belief-state geometry (BSG) of an ideal predictor: the transformer builds a world model in order to predict. We can only see this because we know the ideal BSG in the first place.

Singular learning theory (SLT) links generalisation to effective model complexity, which the Local Learning Coefficient (LLC) estimates: at equal training loss, a lower LLC means a lower Bayesian generalisation error.

Relating the growth of the BSG to the LLC would test an SLT measure of complexity against a world model whose ground truth we know. Calibrated on known processes, the relation would let us infer when transformers learn the BSG of processes whose hidden Markov model we do not know.

Shai et al. watched the belief-state geometry form over training and, in a companion post, asked whether its development relates to phenomena in singular learning theory [1]; the question is still open. In October 2026, Gietelink Oldenziel, a co-author of [1], described the two programmes as largely orthogonal: SLT studies training in weight space, belief geometry studies inference in activation space [12]. Studies of belief geometry over training describe its formation without a measure of complexity. For factored processes, the activations compress, rearrange and plateau in a "representational attractor" [2]; over 100M steps on Mess3, the probe error falls as a power law, then dips [3].

The LLC can be estimated by sampling in networks of this size [4]. In small transformers trained on language and on in-context regression, critical points of the LLC curve, where its rate of change vanishes, mark developmental stages that coincide with new internal structure, such as n-gram statistics and induction heads; the LLC rises in some stages and falls in others [5]. Refined LLCs follow this development head by head [6]. In a toy model of superposition, SGD passes through critical points of increasing LLC [7]. In grokking, the network moves from a higher-LLC memorising basin to a lower-LLC generalising one [8].

None of these studies trains on a process whose belief geometry is known, so none can set the LLC against what the network has learned. The geometry has its own computable complexity: the memory needed to track the beliefs diverges at a rate set by the statistical complexity dimension [9].

First, we would validate the depth measure. Depth-k beliefs, the Bayesian filter on the last k tokens, locate the belief to a level-k cell of the geometry. The cells shrink exponentially, so the plain R² saturates: on Mess3, a representation that encodes only the depth-4 belief already scores R² = 0.98 against the full belief. We would therefore score the increment each level adds, normalised by its variance, on held-out prefixes. Synthetic representations encoding exactly depth K must return K.

Then we would train the paper's transformer on Mess3 and RRXOR, several seeds each, and set the LLC [4] against the resolved depth at log-spaced checkpoints, as many as a pilot of cost and precision allows.

We would place stage boundaries at critical points of the LLC curve [5] and ask four questions:

• 1.Timing: does each critical point mark a new resolved level of the belief state geometry?
• 2.Location: which layers and heads store the new detail (refined LLCs [6]), and is RRXOR's complexity spread across layers like its geometry, which was found only in the concatenation of all layers in [1]?
• 3.Scale: does the final LLC grow with the geometry's statistical complexity, across Mess3-family processes of differing statistical complexity dimension [9] and RRXOR, whose belief set is finite? Two independent Mess3 copies test factoring [2], which PCA can hide; we conjecture their LLC approaches the sum of the factors'.
• 4.Regime: does the LLC track the geometry or only the loss? In our preliminary results, lazy networks (trained with Chizat–Bach output scaling [10]) separate loss from geometry: they predict well while their geometry stays at its initial level. We would report their representation's movement, the probe's gain over initialisation and the excess loss, and calibrate the sampler per α and process, since output scaling multiplies the loss curvature by α².

If compute allows, we would predict each process's resolved depth from its LLC alone, with the relation fit on other processes.

We test three hypotheses.

Association: over training, the LLC tracks the resolved depth of the geometry beyond what the loss detects.

Staircase: each critical point of the LLC coincides with one new level of the fractal, where the loss reaches that of the order-k Markov predictor, by analogy with the toy model of superposition, in which SGD passes through k-gons of increasing LLC [7]; compressions such as the one that precedes the factored representation [2] would appear as falls.

Scale: across processes, the final LLC grows with the statistical complexity dimension. This one is a conjecture: Marzen and Crutchfield tie the number of features near-optimal prediction needs to the fractal dimensions of the belief distribution [11], but no result yet orders network LLCs by these resources.

We would drop the staircase if critical points and new levels do not coincide, and the scale hypothesis if the final LLC tracks the loss or the entropy rate rather than the complexity of the geometry. We would drop the question if the LLC does not track the resolved depth beyond what the loss explains, or if lazy networks, which in our preliminary results keep the geometry at its initial level, trace the same LLC curve as rich ones.

We aim to produce LLC curves over training for Mess3 and RRXOR, and possibly for a factored process, set against the resolved depth of the belief geometry, together with a stage-by-stage map between the two. The level of detail, the number of process variants, seeds and network sizes, and extensions such as refined LLCs per head and the held-out prediction depend on funding. The models are small; the cost lies in estimating the LLC at many checkpoints, with the devinterp library.

Let $R(w) = \frac{\gamma}{2}\|w - w^*\|^2$ be the localization penalty, anchored at a moving point $w^*$. Lin's conjecture: If you do deep learning with this regularization then you are doing internal model selection. For which γ is this true, and what happens otherwise?

A network can solve a task in several ways, each a region of weight space with its own loss $L_\alpha$ and learning coefficient $\lambda_\alpha$, the complexity measure of singular learning theory (SLT). Bayes picks the region with the smallest free energy $nL_\alpha + \lambda_\alpha \log n$, switching to a more complex region as data grows. Those switches are the phase transitions and stages that developmental interpretability reports, using the local learning coefficient (LLC), which is estimated with exactly the penalty R. So R both measures complexity and, if Lin is right, makes training choose regions by free energy. If true, trained networks perform Bayesian model selection internally. If it fails near transitions, where stage claims are made, they are artifacts.

Definitions. Fix a region Ω containing $w^*$, let $w_0 = \arg\min_\Omega L$ and $r = \|w^* - w_0\|$. Let λ(w₀) be the learning coefficient (RLCT of K − K(w₀)) at w₀. Let $F_\gamma(w^*) = -\log \int \exp(-n\beta L_n(w) - R(w))\, \varphi(w)\, dw$ be the localized free energy, and define the finite-n estimate $\hat\lambda(n,\gamma) = n\beta(E_p[L_n] - \min_\Omega L_n)$, the quantity the LLC estimator computes.

Open problem (Lau et al., arXiv 2308.12108, App. D). They write: "we conjecture that $\tilde\lambda(w^*)$ is equal to the LLC $\lambda(w^*)$ given certain conditions on γ … ascertaining the exact conditions on γ [is] left as future theoretical work." Current practice is the heuristic "keep γ between 1 and 10".

A. Prove restricted localization results with explicit errors

B. Use a fully specified polynomial benchmark

Retain

For evidence calculations, take $W=[-1,2]\times[-1,1]$, uniform prior density $1/6$, and fixed regions $U_0=W\cap\{u<1/2\}$ and $U_1=W\cap\{u>1/2\}$.

C.Fall back: Prove the theorem for monomial losses or separable losses (where the integrals factorize into one-dimensional ones).

Estimator window: widest at the crossing, closing past it; selection window: the reverse. Toy: gradient descent splits by basin volume, independent of λ; Langevin and anchored training shift toward the quartic well by the Bayesian gap 0.25 log(nβ). Plain SGD switches later than Bayes.

What would change our mind, and what it would mean. (i) Anchored training matches gradient descent's split under every γ schedule: descent on F_γ never crosses regions, so R measures but does not select, and the conjecture reduces to the known estimator. (ii) Its preference is flat in n or favours the regular well: selection is driven by the O(1) term (extent of the low-loss set), not λ log n, so LLC "stages" track valley geometry, not complexity. (iii) Plain SGD matches Bayes at every learning rate and batch size: degenerate noise is no obstacle, R is idle, and the strong conjecture holds.

Three months: the γ-window theorem with explicit error terms; a γ-selection rule contributed to devinterp; the polynomial and reduced-rank benchmarks comparing anchored training, Langevin and SGD against Bayesian region occupancy; a short paper resolving Lau et al.'s conjecture and stating exactly when "deep learning with the localization penalty is internal model selection".

In a tanh network at an exact realisation with one sextic transverse direction, there is a declared range of inverse temperature and localisation, within a stated budget, where the exact localised coefficient exceeds the Hessian prediction by a persistent 1/6 absent in a regular nonlinear control, and SGLD resolves it.

Terms (not counted). Three quantities are kept apart: λ, the asymptotic learning coefficient (Watanabe's real log canonical threshold [7]); $\Lambda(t, \gamma) = -t\, \partial_t \log Z(t, \gamma) = t\, E[K]$ under the posterior $\propto \exp(-tK - \gamma\|w - w^*\|^2/2)$ on a declared domain, the exact finite-temperature localised observable; and $\hat\Lambda$, what the sampler returns. The Hessian prediction is $G(t, \gamma) = \tfrac{1}{2} \sum_i t h_i/(t h_i + \gamma)$ over the Hessian eigenvalues $h_i$, exact for a quadratic loss. The measured excess decomposes as $\hat\Lambda - G = (\lambda - r/2) + [(\Lambda - \lambda) - (G - r/2)] + (\hat\Lambda - \Lambda)$: the asymptotic geometric excess, the finite-temperature and localisation correction, and the sampling error. "Persistent" means the first term, the part that does not decay as t grows. At smooth points of the zero-loss set it is Kurumadani's (c − r)/(2k) under his nondegeneracy conditions [1, 2], with r the Fisher rank, c the codimension and k the transverse jet order; the sextic case is k = 3, excess 1/6.

Developmental interpretability reads local learning coefficient estimates along training as effective parameter counts, and Timaeus's own guide says the sampler's temperature, localisation and step size have no defaults and must be swept [6]. At smooth points of the zero-loss set, what the coefficient adds to curvature is a definite number with a definite cause, (c − r)/(2k). But a finite-temperature excess over the Hessian prediction is not evidence of it: the regular loss tanh²w gives Λ − G = 1/(2t), a fixed Gaussian localiser lets a distant flatter zero take over as t grows, and a nearly singular εv² term imitates the singular case for γ² ≪ t ≪ ε⁻². Whether a practical measurement can see higher-order geometry at all, in which range and at what cost, is what anyone reading an LLC curve needs to know. A negative result names the constraint that blocks detection.

Kurumadani [1] proves λ = r/2 + (c − r)/(2k) at smooth points of the realisable set under jet nondegeneracy, and his dissertation [2] computes the two-unit tanh case with zero input weights explicitly, (d₁, r, m) = (2, 1, 3), λ = 2/3 under a₁² ≠ a₂², with the four-unit extension; the asymptotic answer for the test case is settled, which is what makes it usable as ground truth. Wang et al. [3], Appendix D.3, state λ ≥ r/2 and find Hessian-rank estimates unreliable. Cai [4] derives continuous-time Gaussian-localised targets for u² and u²v² (a Bessel-function closed form for the centred product singularity), separates them from finite-step Langevin effects where a closed form exists, and gives the mean energy under a Gaussian surrogate (his Proposition 1), all in service of a thermodynamic-variance estimator on MLP trajectories; he has no regular nonlinear control, no smooth-fibre sextic geometry and no network prediction. Hitchcock and Hoogland [5] benchmark sampler fidelity to the local posterior. Nobody has written down a diagnostic that separates a persistent excess from the O(1/t) correction of a regular nonlinear minimum, derived the visibility scale $t \asymp \gamma^k/a$ with the coefficient and localisation metric taken in a network's own coordinates, or tested whether an actual tanh network's 1/6 can be resolved within a budget. That is the gap.

First, the benchmark. Four losses with the same Hessian diag(2, 0) at the origin: u² with an unused v (λ = 1/2, Gaussian calibration); tanh²u with an unused v (λ = 1/2, the regular nonlinear control, Λ − G = 1/(2t)); u² + a v⁴ (λ = 3/4, persistent excess 1/4); u² + b v⁶ (λ = 2/3, excess 1/6). Kept separate: $K_\varepsilon = u^2 + \varepsilon v^2 + v^4$, whose quartic regime γ² ≪ t ≪ ε⁻² exists only when εγ ≪ 1, as a resolution test; and u²v², with its 1/2 − 1/log ρ approach, as a slow-convergence test. For each, compute Λ(t, γ) by quadrature on a declared domain, localiser imposed in the stated coordinates, and G on the same domain.

Second, the diagnostic, fixed before any network result. Over a declared t-range, fit Λ − G to A + B/t + C/log t; the excess is persistent when A separates from zero at the stated resolution while the tanh² control returns A = 0. Quantify the minimum detectable curvature $\varepsilon_{\min}$ from the $K_\varepsilon$ window. Then run SGLD at the same (t, γ), varying step size and chain length separately, to measure $\hat\Lambda - \Lambda$ and the cost of resolving 1/6.

Third, the network. Two students with zero truth: the two-unit network at (a₁, a₂, b₁, b₂) = (1, 2, 0, 0), and a four-unit student made of a fixed nonzero two-unit teacher plus a zero pair, (r, c, k) = (5, 6, 3), λ = 8/3, teacher units free. Evaluate the actual loss in the original coordinates with the localiser imposed there; derive the sextic coefficient (1/700 before KL normalisation) and the induced localisation metric; predict the visibility range t ≍ γ³/a and its cost; then test the prediction with the diagnostic of step two.

Fourth, deferred until the first three work: one training trajectory with the checkpoint-generated divergence $D(p_{w_t} \,\|\, p_w)$ as the loss, stated as such.

Prediction. On u² and tanh² the diagnostic returns A = 0, with B = 1/2 for tanh²; on the quartic and sextic controls it returns A = 1/4 and 1/6 once $t \gg \gamma^k/a$; $K_\varepsilon$ returns A = 1/4 inside its window and 0 outside; u²v² approaches 1/2 slowly enough that a short t-range misreads it. In the two-unit network the predicted visibility range is nonempty at laptop cost and SGLD returns A = 1/6 within the stated resolution; the four-unit student does the same at higher cost.

What would make me drop it. If the predicted visibility range is empty within the budget, or the diagnostic cannot separate the sextic control from tanh² over any affordable t-range, then higher-order geometry at this order is not measurable with this workflow in this family. The report then names which constraint blocks detection, locality, the finite-temperature correction or the sampler, and how it scales with k and the coefficient. That is a result, not a failure, and it licenses no claim about published LLC curves in other models.

Two weeks give the benchmark, the diagnostic with its false-positive check, and the two-unit network prediction with one test of it. That is the decision point. Three months, only if the two-week criterion is met: the four-unit student, the visibility-and-cost result written as a short paper, and a second geometry (a coinciding pair, k = 2) for the scaling in k. Needs a laptop, the quadrature code from the first proposal's pilot, and devinterp. Drop first the four-unit student, then $K_\varepsilon$. Postponed as separate problems: trajectories, the hidden-noise limit of the layer-composition bound, and complex thresholds of fibre ideals.

Can certifying that a monitor can match an LLM’s use of its written reasoning require exponentially many tests, even in a simplified model where each test measures both systems’ best performance exactly and their abilities without the reasoning match?

Suppose an LLM writes reasoning that helps it solve a problem. A trusted monitor reads the same text and performs equally well across many evaluations. Can we conclude that the LLM has no important use for that reasoning the monitor misses?

If ruling out this advantage requires exponentially many tests, accurate evaluations alone cannot provide a general guarantee. We would need additional assumptions about how the models use their reasoning, or access beyond evaluation scores.

If a small, systematically chosen set of tests suffices, it could give auditors a justified stopping rule and guarantees covering untested objectives. Either outcome would clarify what passing evaluations actually establishes. Applying either conclusion to real LLMs requires checking that they fit the mathematical assumptions.

Anwar et al. [1] propose detecting hidden information by measuring how much a reasoning trace improves an LLM’s performance compared with a trusted monitor’s. They show that a fixed collection of tasks can miss information useful for another task. They do not determine how many tests are needed when the underlying situations stay fixed and the auditor chooses each new test using earlier results.

Blackwell [2] connects the usefulness of information across all decision problems with the ability to reproduce one information source from another. Le Cam [3] develops approximate versions of this comparison. These results explain why checking every relevant objective can establish a meaningful guarantee. They do not directly tell us how many selected tests suffice for agents with restricted abilities.

There is also a geometric connection: each agent’s possible behaviors form a shape, and a test measures how far that shape extends in one direction. Eisenbrand et al. [4] study the difficulty of determining whether one shape fits inside another, under different access and error assumptions.

The first research task is to check whether those existing results already settle our monitoring question. The project is worthwhile only if the monitoring assumptions leave a substantive gap.

First, construct a small example with two equally likely situations and two possible actions. Give the model and monitor the same choices without a reasoning trace, but let the model use the trace to choose its action differently in each situation. Calculate which tests reveal this advantage and which miss it. This gives a concrete starting point for checking the definitions.

Next, use the Blackwell connection to define what successful monitoring should guarantee: for each model strategy, the monitor can reproduce approximately the same pattern of actions across situations. Matching overall action frequencies is insufficient; the actions must occur in the right circumstances.

Then fix the auditor’s access. It chooses scoring rules and receives both systems’ exact best scores, but cannot inspect their internal rules. Its next test can depend on previous answers. Keeping the situations and their probabilities fixed separates test selection from changes in the environment.

Build larger examples in which a substantial advantage is revealed by only a small fraction of possible tests. Try to prove that finding it requires exponentially many tests as the number of situation–action pairs grows. This is not a claim about exponential growth in model size.

Compare the construction with known geometric results before claiming novelty. Finally, look for assumptions that make certification easier, beginning with a model whose available strategies are explicitly listed. These comparisons would identify which restrictions create the difficulty and which additional information removes it.

I expect some model–monitor pairs to agree on many tests while differing substantially on an objective those tests miss. Even choosing tests in response to earlier results may fail to uncover the difference quickly. This is a prediction about the simplified setting, not evidence that real LLMs routinely hide such advantages.

I expect certification to become easier when the auditor knows a short list covering the model’s possible strategies. That would give a concrete example of additional knowledge making evaluation results more informative.

A method that reliably detects every substantial advantage with only polynomially many tests would refute the proposed exponential requirement. I would then investigate that positive result. I would drop the project as a standalone contribution if existing geometry already settles the question and no meaningful monitoring restriction remains unresolved. I would reconsider its LLM relevance if the difficult examples depend entirely on implausible restrictions on the monitor.

Three months would produce a precise model, worked examples, and a proof that many tests are necessary or a method showing that fewer suffice. The first month would check existing results and build examples; the second would develop the main argument; the third would examine one easier case and write the findings.

The work needs a laptop, optimization software, and advice from researchers in decision theory and geometry. It does not initially require large-model experiments. If time becomes tight, I would drop extensions involving noisy scores or changing environments and keep the question about exact tests in a fixed setting.

Can models learn, from realistic RL training setups, to write steganographic notes to themselves? I’m specifically interested in demonstrating that this is feasible.

I think models are clearly able to encode hidden messages in their outputs (cf. classic subliminal learning setup). As we rely more and more on agentic coding, where models leave many notes for themselves (Claude’s briefs, Anthropic’s ‘git for agents’, etc.), it’s crucial for safety that we understand what’s being communicated in those notes. Sufficiently powerful models could also use general steganographic capabilities to pursue other misaligned goals (e.g., exfil), and this would greatly weaken our monitoring defenses. If it can be shown that a reasonably realistic RL setup could lead capable models to develop such capabilities, labs will hopefully be pressured to rethink and tighten their safety measures.

Training against CoT monitors makes models obfuscate their reward hacking, but this happens only within a single context [1]. Penalizing specific strings in the CoT makes models encode them [2]. Steganographic collusion has also been demonstrated [3, 4]. As far as I know, no one has studied whether models can learn, and generalize, to produce benign-looking steganographic text that only the same model can understand, with no shared context.

I would run RL in a carefully designed environment. In particular, each rollout would consist of two phases, with context flushed in between:

Exploration: the agent is given the task context and an opportunity to look around the environment, where a specific reward hack is implanted. It’s also told that it will have to solve the task in a second phase, with a fresh context but a shared scratchpad, and that it can leave notes for itself there.

In the second phase, the model is encouraged to read its scratchpad and conclude its work. Depending on how experiments go, we could remove the hack from the environment in this phase.

The reward includes a penalty term from a weaker monitor model. The monitor has access to the scratchpad but not the CoT. Crucially, we can make the RL task impossible to solve without the hack, so that the model would be pressured to encode it in its scratchpad. Optionally, before RL, I would run SDF on documents describing steganography as a way to evade monitors. I think this setup, while synthetic, is not that far from real-life agentic RL environments.

I expect weaker models to likely fail to learn anything meaningful.

With more capable models, I think the right optimization pressure is there, and I’m genuinely uncertain if they’ll learn.

The nature of the hack is also important. If it’s something syntactic and simple, it could learn a trivial encoding that’s not practical for any actual use cases. It should ideally be something semantically rich.

Hopefully, whatever steganography it learns would be obvious from reading its CoT.

Most interesting / worst-case scenario: it gets the reward with no obvious signs of reward hacking, both in its scratchpad or CoT. Steganography generalized to CoT.

Three months should be plenty provided enough compute. I would probably start with a setup that’s most likely to succeed (a lot of cues, easy to find hack, etc) and see what’s the smallest model that learns anything. One cue could be continued pretraining on steganographic text. If I get a working setup, I would try to find a minimal version and test if these steganographic capabilities generalize beyond the RL task (e.g., to standard agentic coding).

When two AI systems hide information in harmless choices about how they express the same reasoning, we conjecture that, in a simple model of structured rewriting, eliminating covert communication requires changing essentially the entire message.

Language gives us enormous freedom to express the same idea: we can change words, examples, ordering, or the structure of an explanation. Two AI systems could use these harmless choices to communicate information that a monitor does not notice.

A natural fix is paraphrasing: rewrite one model’s reasoning before another model sees it. The basic quantitative question is: how much rewriting is needed to reduce the maximum possible rate of hidden communication?

If a modest amount of paraphrasing can drive this rate to zero, paraphrasing could be a strong general defense. However, there is a natural tension between preserving useful reasoning and preventing covert communication.

Zhou and May [1] show that language models can hide information in high-level reasoning choices rather than particular words, and that these signals can survive ordinary paraphrasing. A paraphraser that knows the particular hidden strategy performs substantially better.

Rozinek [2] comes closest to a mathematical treatment. It separates the intended meaning of a message from the freedom in how that meaning is expressed and studies the information that survives a fixed paraphrasing channel. However, it does not characterize the full tradeoff between how much rewriting is allowed and how much covert communication remains.

We study this tradeoff first in a setting where “same meaning” is exact. Consider different ways of grouping a sequence of statements joined by “and.” For example,

(A and B) and C

and

A and (B and C)

have exactly the same meaning. A local regrouping changes one into the other.

These groupings correspond to binary trees. Sleator, Tarjan and Thurston [3], and later Pournin [4], studied how many local regroupings may be needed to transform one tree into another. Our question is different: what is the smallest set of representative groupings such that every possible grouping can be transformed into one of them within the rewrite budget? Even the best universal paraphraser must leave that many distinguishable outputs, which determines how much hidden information can still be transmitted.

We begin with n statements joined by “and.” Different groupings have exactly the same public meaning, so Alice and Bob can use the choice of grouping to encode a hidden message.

The paraphraser is deterministic. It sees the whole input and may make at most k local regroupings. It must commit to its rule before Alice and Bob choose their hidden code. Alice and Bob know the rule and may choose whichever code survives it best. This models a general-purpose defense rather than one designed against a known code.

We first require zero-error communication: Bob must always recover the hidden message. For deterministic paraphrasers, this gives the same asymptotic rate as allowing the decoding error to vanish.

The problem then becomes a covering problem. Let $K_n(k)$ be the smallest number of representative groupings needed so that every possible grouping is within k local regroupings of one representative. The surviving covert rate is

where δ is the fraction of the structure that the paraphraser may change.

We have two starting bounds. A direct neighborhood-counting argument gives g(δ) ≥ 2 − H₂(δ) − 2δ > 0 for δ < 1/2, where H₂ is binary entropy. Conversely, every grouping of n statements can be transformed into the same fully right-grouped expression in at most n − 2 local regroupings [5, Proposition 4.3], so g(1) = 0. The zero-rate threshold therefore lies between 1/2 and 1. We conjecture that it is 1: even allowing a subexponential number of representative outputs cannot improve the leading constant in the rewrite budget. We would attack this conjecture by bounding neighborhood sizes and constructing small covers.

We would also compute the optimal covering numbers for small n. This should reveal the shape of g(δ), identify unusually easy or difficult groupings, and suggest the right conjecture before attempting the asymptotic proof.

Finally, we would test whether the same qualitative phenomenon appears in real LLM reasoning. One model would solve benign reasoning tasks while encoding a controlled hidden message through choices such as the order of independent steps, choice of example, or choice between valid solution paths. A paraphraser would then rewrite the solution at different strengths, and a second model would attempt to recover the hidden message. Here paraphrasing strength means the number of allowed local edits, reported as a fraction of the steps or sentences changed.

For each paraphrasing strength we would measure two quantities separately: how much hidden information remains recoverable, and how much ordinary task performance is preserved. This gives an empirical analogue of the theoretical rate-versus-rewriting curve.

We expect g(δ) to decrease as the paraphraser is allowed to make more changes, but to remain positive until the rewrite budget is very large. Our stronger prediction is δ* = 1: eliminating every positive covert rate requires (1 − o(1))n local regroupings, which is essentially the entire structure.

If g(δ) = 0 for some δ < 1, so that a budget of δn regroupings already allows $K_n = 2^{o(n)}$ representatives, then the conjecture is false and universal paraphrasing is more powerful than we expect. We would then try to determine the true threshold and the full shape of g(δ).

For the LLM experiment, we expect generic paraphrasing to reduce hidden communication gradually rather than eliminate it immediately. We would reconsider the relevance of the toy model if mild paraphrasing consistently destroys structural hidden signals while leaving task performance essentially unchanged.

In three months, the main theoretical goal is to obtain a new bound on g(δ), or at least substantially narrow the location of the zero-rate threshold. We would combine proofs with exact computation on small instances.

A secondary goal is one controlled LLM experiment measuring hidden communication rate and task performance as paraphrasing becomes stronger. This is meant as a test of whether the theoretical abstraction captures a phenomenon visible in real models, not as a full empirical study.

If time is limited, we would prioritize the theoretical result and small-instance computation, and drop the LLM experiment first.

Can a backdoor be planted in a transformer’s embedding or attention weights so that, after a specified training process, the full weights are computationally indistinguishable from those of an honestly-trained model, while concealing the backdoor in its weights and hiding a secret trigger in its text?

Assessment of a model’s trustworthiness uses access to weights, behavioral evals, and inspection of the inputs. A backdoor (in which an adversary directly manipulates the outputs of a model, without the user’s knowledge) embedded in the trained attention would test the limits of all three: it could preserve performance, but suddenly activate on text that appears statistically ordinary. The classical example of using sparse PCA for the weights to “trigger” the hidden backdoor in ReLU is one version of the backdoor. Our project will ask whether these properties can be proven together under a training process for transformers; this will combine cryptographic indistinguishability, information-theoretic guarantees for text distributions, and the actual structure necessary in attention that identifies what assumptions such a backdoor would need. Negative results are also important: showing that training destroys such a mechanism, or that an efficient auditor can detect it, would identify concrete limits for the “hidden” model's behavior.

Existing results establish different parts of this goal, and use different meanings for “undetectable”. The proposal will be to find hiding guarantees and a functioning backdoor in one trained attention model. [8] gives the original foundational construction of undetectable backdoors, including white-box results for a certain random-feature model. [1] further obtain statistical indistinguishability results for frozen Gaussian first layers and a sufficiently Lipschitz downstream network. Neither works establish how attention computationally acts when the relevant weights are trained, or how they can be activated by “statistically ordinary token strings”. [2] combines hidden backdoors with steganographic activation, with no reference to obfuscation of the model. [3,4] address a related question: can it be computationally hard to discover an input that activates the hidden behavior, which is technically different from being statistically indistinguishable from the weights. A circuit can be hard to activate, while its structure looks recognizable. [6] provides a route that preserves a covertext distribution and does not automatically supply a decoder inside trained attention.

We will investigate whether the hidden-weight mechanism in [1] can be adapted to trained transformer attention. The aim is to modify one or several heads so that a secret trigger introduces a signal into the residual stream that produces a chosen incorrect response.

First, we will specify the architecture, initialization, training data, optimizer, training budget, and text generator. These define the honest baseline. We will compare distributions of released weights, seeking computational indistinguishability and, where possible, stronger total variation bounds.

Next, we will build a small attention model with a visible backdoor. The trigger will redirect attention from task information toward an incorrect answer. This will identify the attention scores and output margins required for activation, without claiming either hiding guarantee.

We will then investigate hidden relations among embedding or attention weights. The Gaussian-layer construction and bi-Lipschitz assumptions in [1] provide starting points, but require new arguments for attention due to softmax coupling across tokens and learned query-key structure. Honest and planted models will undergo identical continued clean training. We will track embeddings, attention projections, and predictions: preserving a planted coefficient does not ensure that its effect survives.

Finally, we will develop an encoder whose valid token strings activate the mechanism while statistically matching honest model output. The initial guarantee will cover one sample, averaged over model and secret randomness.

Experiments will measure clean performance, triggered errors, accidental activation, and attention margins. Detection tests can reject candidates, but their failure cannot prove hiding. Success requires distributional guarantees and surviving behavior in the same construction under consistent training and sampling assumptions.

The strongest outcome is a single construction satisfying all four guarantees under a fully specified training process, with a proof of the weight comparison, text comparison, and surviving behavior. A restricted attention architecture would be a meaningful first setting if its restrictions are explicit and both hiding guarantees concern the same construction. This would help with monitoring, controls, and detection for AI safety as well, and we could pivot to an experimental procedure instead.

Our working hypothesis is that training can preserve a sufficiently separated attention mechanism when clean data exert little gradient in the secret direction. This is a hypothesis to test. The toy shows why weak gradients in that direction alone are insufficient: other parameters can move enough to erase activation.

We would change direction if the candidate admits an efficient weight test, its triggering states cannot be reached by valid tokens, or ordinary clean training removes the behavior in the intended regime. A rigorous result establishing one of these failures for an identified class of constructions would be useful. Failure would imply robustness of transformers to obfuscation in general, at least a weak sense.

Within three months, we aim to produce a precise problem statement, a construction or substantive obstruction for one small attention setting, and reproducible experiments. The first month will establish the baseline and transparent mechanism; the second will investigate weight hiding and training dynamics; the third will address text encoding and consolidate results. Understanding exactly how backdoors can manipulate features in the residual stream at a mechanistic level is not obvious and should be explored.

We need theoretical guidance in cryptography and optimization, plus modest GPU access for small models. If integration exceeds this timeframe, we will prioritize the attention and training theorem, explicitly leaving any unresolved text encoding requirement for subsequent work.

On Python benchmark solutions, does rewriting a model's code into one fixed standard form, or into a random equivalent form, cut the hidden bits a decoder recovers from its logical structure to near zero, while the code passes its tests?

• •Setting. In untrusted monitoring, one model copy writes code and another checks it. A shared hidden signal lets backdoors through; one bit suffices.
• •Gap. Paraphrasing removes signals in wording, not logic: which checks code makes, how loops, guards and edge cases are handled.
• •Two model-free sanitisers:

o Canonicalise: map equivalent programs to one.

o Perturb: pick a random equivalent, with a secret seed. Rewrites can be undone, so this never beats a standard form on information; it covers choices too large or complex to standardise.

• •If it works: a model-free defence that cannot be talked out of its job.
• •If not: we learn which logic carries messages, and where monitors should look.

• •Paraphrasing, in text. Paraphrasing cuts simple hiding schemes to under two bits per answer [1]. Bits in high-level reasoning patterns resist a strong paraphraser [2].
• •In code. Gardner-Challis et al. [3] hid a signal in “for” versus “while”. Renaming and reformatting barely helped on BigCodeBench (safety 6.7% to 13.3%). A model-written summary reached 91.7% but hid details the monitor needed. Their minifier never changes control flow, so the middle ground is untested.
• •Adaptive attackers. Learned collusion still emerged under paraphrasing, and a code paraphraser let one bit through over 99% of the time [4]. Randomised defences elsewhere fell once attackers averaged over the randomness [8]; code watermarks with error correction survive random edits [9], so rewrites must reach the program's structure.
• •Neighbouring fields. Network "active wardens" reset every free packet field to a default [10]. Plagiarism detectors undo statement reordering, but not against hidden signals [11].
• •Theory. Rice's theorem rules out a perfect standard form for every program, so the work is finding forms that remove the most hiding capacity.
• •What is missing. We found no work that rewrites code's logical structure without a language model, or measures which hiding channels survive.

• 1.Benchmarks and encoder.
• •BigCodeBench [5] and APPS [6], as in [3], so results compare. HumanEval+ and MBPP+ [7], whose larger test suites leave fewer untested inputs.
• •A frontier model hides bits using a codebook over four channels: names; structure (loop form, guard style, statement order); algorithm choice; edge-case behaviour. A decoder with the codebook reads them back.
• 2.Build the sanitiser.
• •LibCST rewrites at a catalogue of choice sites, each with a safety condition; egglog [12] for typed pure expressions.
• •One switch picks the canonical or a seeded random choice at each site. The rewritten program is the one that runs.
• •Every output passes the tests and differential fuzzing [13]. Code we cannot handle goes to the trusted monitor, and that refusal counts as a possible signal.
• 3.Measure. Each defence is a channel from hidden message to rewritten code. A decoder with bit error rate e gives 1 − H(e) bits per hidden bit, a lower bound, so we also train a classifier as a stronger decoder. Defences: none, renaming and reformatting [3], language-model paraphrasing [1], canonical, random.
• 4.Adapt. The encoder is told the defence and uses error-correcting codes. Channels that keep positive capacity are the main result; we test how much stronger tests close them.
• 5.Control check. Is what survives enough to collude? The “for”/“while” game of [3] on APPS, and Veron's trained colluding monitor [14]: signalled backdoors caught at a fixed false-alarm rate, and the cost to honest detection.

Expect:

• •Canonicalisation drives structural channels (loop form, guard style, statement order) to near zero.
• •Random perturbation leaves a small capacity that error-correcting codes exploit.
• •Both keep pass rates above 95%.
• •Algorithm choice, such as recursion versus iteration, partly survives both.
• •Edge-case behaviour and tie-breaking survive every behaviour-preserving defence, carry most of what remains, and shrink as tests get stronger.
• •In the control check, at most one bit survives, and honest detection stays within five points of no defence.

Would drop the question if:

• •language-model paraphrasing already drives structural capacity to near zero, so a model-free sanitiser adds little;
• •if canonicalisation cannot keep pass rates above 90%, or is often intractable, we focus on the random perturbation approach instead;

Three months produce:

• •a set of program pairs that differ in only one hiding channel, such as loop style, algorithm choice or edge-case handling;
• •an open-source tool that rewrites Python into a fixed standard form or a random equivalent one, checking every rewrite with tests and fuzzing;
• •the capacity of each channel under each defence, in bits per solution;
• •the control-check result.

Needs: API credit for the encoder and paraphraser, a few GPU-hours for the classifier, and a mentor in AI control or programming languages.

Drop first: the HumanEval+/MBPP+ test-strength study, then BigCodeBench.

The models of the same family can sometimes “read each other's thoughts”. We want to study the extent to which models can leverage this acausal coordination (tacit collusion), to transmit complex messages using steganography.

This is important because we can understand how two copies of the same model could conjure up the same steganographic key, thus bypassing the need to transmit it. The key in the steganographic sense can be understood via key-picking and encoding procedure, which can implicitly be represented in the model so that the models can encode/decode messages "subconsciously".

Beyond its relevance to backdooring, this is important for our conceptual modeling of AI agents. For example, applicability of evidential decision theory to agent interactions depends on their ability to communicate tacitly.

There is previous work that attempts to answer parts of this. For instance:

• •In [1], this was done by asking models to play coordination games, such as picking a number from 1 to 100. However, since the models have the same prompts, models of the same type would immediately have the same output logit distribution, so copies agreeing is guaranteed rather than evidence of a shared private key.
• •[2] has a Sender write descriptions of two words that secretly mark one as the target, and an isolated Receiver pick the target. Same-model pairs beat cross-model pairs. However, the descriptions are a message, so this tests decoding a signal.
• •[3] names acausal Schelling-point collusion as a threat to untrusted monitoring and bounds passive self-recognition by fine-tuning GPT-4.1. However, it only studies one model.
• •[4] had two copies of Claude 3.7 Sonnet write a policy and a monitor for a coding task, and found rare collusion on "magic numbers" such as 42 and 1337. Similar to [1], the two prompts were identical apart from the role line.

We start by developing a benchmark that measures models’ abilities to collude/share secret messages across models of the same type and different types. This is done as proof of concept today as well as eyeing the future where the models are more capable.

Next, we will use interpretability techniques to identify mechanistic traces of such self-collisions and intervene accordingly to reduce collusion abilities. This is done perhaps by first looking for “shared decoding key” direction in the activation space, or potentially in the weights itself and see if steering/weight-based ablation techniques would work to lower the collusion abilities.

We believe that models of the same type will have higher ability to collude (i.e. be able to decode secret messages better). This is because we have seen success in “similar architecture” interpretability techniques, such as subliminal learning [5], activation oracles [6], and training models to introspect [7].

We also believe that there should be a mechanistic way to describe and understand the shared key direction/subspace in the weights of the models themselves. This is because if the empirical results (from the first paragraph) is true, combined with the fact that models of the same type have the same architecture, then there might be some sort of mechanistically “shared structure” that helps to explain this phenomenon of tacit collusion/acausal coordination.

If either of these steps fail (i.e lack of ability of collude/interpretability methods does not work as expected), we will then rephrase the question to see why this fails.

• 1.An evaluation benchmark.
• 2.Formalization of what it means to be able to tacitly collude/send secret messages. (This can be done through a behavioral definition, potentially validated through interpretability lens.)
• 3.We need some compute cost to run multiple experiments (ie API/running interpretability on GPUs).
• 4.Mentors with shared interest to bounce ideas off.

Can a weaker LLM, equipped with counterfactual probing and structured decoding, recover hidden messages generated by a stronger adversarial LLM at least 15% points more accurately than a compute-matched baseline, even when the encoding strategy was not seen during training?

CoT monitoring is used to detect unsafe behavior. However, a sufficiently capable model might learn to hide information in its reasoning, making it difficult for a weaker monitor to understand.

Our goal is to investigate whether we can compensate for differences in model capability using better algorithms, not larger models. If a weaker model can recover hidden information through structured probing, we could make monitoring more reliable without requiring the monitor to be as capable as who it supervises.

If this fails, it would suggest that some forms of encoded reasoning create a capability gap that additional inference computation cannot overcome.

Understanding this limitation would help determine when weaker models are sufficient and when stronger monitors/alternative defenses are necessary.

• [1]demonstrated that LLMs can learn encoded reasoning and investigated paraphrasing as defense. Their work focuses on preventing hidden communication rather than recovering the information being communicated.
• [2]introduce the steganographic gap, measuring differences in usable information between a Receiver and a Sentinel. They show that additional monitoring computation can reduce this gap, but do not develop an algorithm for discovering previously unknown encoding schemes.
• [3]investigate the steganographic capabilities of frontier LLMs. They find that models struggle to hide information under standard conditions, but can succeed with additional affordances, including private reasoning and coordination on encoding strategies.
• [4]study how modifying CoT changes model behavior. This establishes counterfactual intervention as a technique for understanding reasoning, but they do not apply it to decoding messages.

More recently, [5] show that steganographic models can evade activation-based detectors and that targeted changes to the evaluation distribution can restore detectability. Their work suggests interventions can reveal hidden information, but does not study adaptive black-box decoding.

Cryptographic constructions [6] also show that keyless detection cannot work universally. Therefore, we focus on learned, noncryptographic encoding strategies.

The open question is whether counterfactual queries can reveal enough structure for a weaker model to recover hidden information.

We would first construct a controlled experiment where a stronger LLM is instructed to hide an 8-bit message inside an apparently ordinary reasoning trace. An intended Receiver, which knows the encoding scheme, must successfully recover the message, while a weaker monitor does not know the scheme.

We would start with Qwen3-32B as the encoder and Qwen3-8B as the monitor, using lexical, syntactic, and semantic encoding strategies. The encoder would generate multiple candidates and select those that evade a baseline monitor while remaining decodable.

Our main intervention would be counterfactual probing. Instead of asking the weaker model to decode the message directly, we would let it modify parts of the reasoning trace and observe how those modifications change the Receiver's downstream behavior.

For example, the monitor could paraphrase half of the sentences and test whether the Receiver still benefits from the trace. If the benefit disappears, the modified sentences may contain information essential to decoding. Repeating this process could help localize the hidden information.

The monitor would then use an LLM to propose candidate decoding rules and small programs, which it could test against calibration examples with known hidden messages. The Receiver-query interface would expose only constrained downstream behavior, not a direct request to reveal the secret.

We would compare this algorithm with direct prompting, few-shot prompting, and additional reasoning tokens, using matched computational budgets. Evaluation would measure exact secret recovery, false-positive rates, and oracle-query complexity on previously unseen encoding strategies.

Finally, we would study a restricted theoretical model where only k of n trace segments carry essential information. Under a reliable, monotone group-testing oracle, these segments can be localized in $O(k\log n)$ queries. We would test when real LLM encodings approximately satisfy these assumptions and when distributed encodings violate them.

We expect that a naive weaker LLM will struggle to decode hidden messages generated by a stronger model, especially when the encoding strategy is unfamiliar. However, we predict that counterfactual probing will improve recovery because the monitor can learn which parts of the text affect behavior.

We expect this advantage to be largest for compositional encoding strategies, where hidden information is distributed across several sentences but is patterned. We expect that structured probing will outperform simply giving the monitor more reasoning tokens.

Our minds change if the improvement disappears when controlling for computation, example access, or encoding strategy. In particular, if counterfactual probing consistently fails on held-out schemes while direct prompting or ordinary fine-tuning succeeds, the proposed intervention would not offer a meaningful advantage.

A second negative result would be finding that the hidden information is too distributed for localization to help, even when the intended Receiver decodes it reliably.

In three months, we aim to produce a reproducible benchmark of adversarially generated hidden reasoning, implement counterfactual probing and structured decoding, and evaluate their performance against compute-matched baselines.

The main deliverable would be an empirical study measuring how secret recovery depends on model capability, encoding complexity, and the number of counterfactual queries. We would also aim for a small theoretical result characterizing when group-testing-style localization is possible.

We would need access to two open-weight LLMs, inference compute for adversarial data generation, and resources for evaluating many interventions.

If necessary, we would drop fine-tuning and focus on prompt-generated-encodings and black-box decoding.

Does cumulant propagation retain its, accuracy scaling with width, when Gaussian-initialized networks receive low-rank weight perturbations?

Wu et al. [1] give an alternative to naive Monte Carlo sampling called cumulant propagation. My aim is to extend the applicability of their algorithm to more realistic training scenarios. Cumulant propagation estimates a network’s expected output by tracking distributional statistics through its layers. It can outperform sampling asymptotically for Gaussian random networks. Useful networks, however, acquire structure through training, precisely where the current guarantee stops applying. We want to know whether cumulant propagation remains accurate after we change a randomly initialized network’s weights. We start with changes chosen independently of the initial weights because these are easier to study than changes produced by training. If the method still works, we can then test whether it also works when training chooses the changes. If it fails, understanding which changes cause the failure could help us improve the method or explain when it can be trusted.

Either outcome would help us move from understanding random networks toward understanding trained networks.

Wu et al. [1] prove mean-estimation guarantees averaged over independent Gaussian weights, for fixed depth and polynomial activations. They also report ReLU experiments and explicitly suggest decomposing weights into structured and random components. They do not establish the corresponding error bound. LoRA [2] motivates studying low-rank updates, but its pretrained base differs substantially from our Gaussian base.

Consider width-n, zero-bias MLPs with standard Gaussian inputs and fixed activation polynomials. Write

The candidate claim is

Here $L$ counts linear layers; rank, norm, depth, and activation polynomials stay fixed as width grows. Each network’s input expectation is estimated first; estimation error is then averaged over Gaussian initializations.

If we choose a fixed weight change before seeing the random initialization, we are just adding fixed numbers to independently sampled weights. The weights remain independent, although their average values change.

Training is different: it chooses changes based on the network’s current weights and behavior. Those changes can make different weights depend on one another.

Our experiments test only the first situation, using ReLU networks and one particular family of fixed changes. The results are encouraging, but they do not yet show that the method works after training. This is a motivating future direction to study.

We tested 1,792 networks at seven widths between 8 and 64. At each width, we compared 128 random initializations with and without the same small low-rank weight change. All networks used ReLU activations and three linear layers.

The estimation error decreased with width at approximately the expected rate. Doubling the width reduced mean squared error by roughly four times for order-2 cumulant propagation and eight times for order-3. At width 64, adding the weight change increased average error by less than 1% for both orders. At width 8, the order-3 error increased by about 14%, suggesting that narrower networks can be more sensitive.

We used repeated sampling to estimate each network’s true expected output, accounted for uncertainty in those estimates, and checked the results using more samples and a different sampling method. These checks supported the overall findings.

The results suggest that cumulant propagation remains accurate under the small, fixed weight changes we tested. They do not yet show whether this holds for stronger changes, different directions of change, deeper networks, or changes learned through training.

Separate weak updates from benign geometry. In the first two weeks, retain the existing estimator and paired-seed measurement pipeline. Compare dense, coordinate-concentrated, and cross-layer-aligned updates with identical singular values. Begin with rank 2, depth 3, and norms 0.25, 0.5, and 1; vary several independently chosen orientations. Measure scaled error, output variance, and the change in the network’s input-averaged output. This checks whether apparent robustness merely reflects updates that barely affect the function.

Use the differences to choose the theorem. If orientation changes the width trend, identify the first affected layer and omitted cumulant. Start the proof attempt with rank-one, shallow polynomial networks, expanding Gaussian and deterministic contributions separately. Exact Gaussian moments in small polynomial examples provide checks. A counterexample would replace the unrestricted claim with a condition on singular-vector concentration or alignment. If no geometry effect emerges, vary rank and depth separately before widening the claim.

Test dependence introduced by training. Freeze the Gaussian base and train low-rank adapters on a synthetic teacher task. Compare learned adapters with orientation-randomized adapters preserving their singular values, and with independently constructed adapters whose spectra were fixed before initialization. Use repeated initialization and data seeds; record task loss and adapter norms throughout. Randomizing learned orientations retains dependence through the learned singular values, so it tests alignment rather than complete independence from initialization.

Change the algorithm only after locating a failure. A reproducible error increase tied to particular directions would motivate explicitly tracking their projected cumulants. Test that modification against the unchanged estimator at matched error and computational cost. Preselect evaluation widths and seed budgets, retain high-error networks, and repeat reference audits for any apparent failure.

I expect small weight changes spread across many neurons to preserve the observed decrease in error with width. Changes concentrated on a few neurons may behave differently. Success would mean proving when this accuracy is preserved, or finding a clear counterexample that shows why the claim needs narrower conditions.

Worse results at larger widths would motivate further investigation, but would not alone disprove the mathematical claim.

Our small training experiment found slower error reduction in both trained and untrained comparisons, so we cannot attribute it to training. If larger experiments also find no meaningful difference, I would focus on understanding fixed weight changes rather than developing a training-specific algorithm.

Risks and open questions

The perturbations may simply be too weak. Fixed rank and norm limit their total energy as width grows. Stronger norm settings and measured changes in the network function will distinguish this explanation from robustness to substantial structure. Nevertheless, these are the right things to focus on at the beginning.

Training comparisons may remain confounded. Matching rank and norm does not match the full spectrum, orientation, or task loss. Report these quantities and interpret each control narrowly; a gap alone does not identify its cause.

Three months should produce one carefully resolved result: a restricted polynomial error bound or explicit obstruction, supported by reproducible experiments and a controlled adapter-training comparison. Month one tests geometry and strength; month two develops the polynomial argument; month three evaluates learned adapters and writes the results. The existing parallel runner and reference audits reduce setup time. Required resources are a multicore workstation, occasional GPU access, and mentorship in cumulant expansions. If the proof or controls take longer, drop broad training sweeps and extra depths first. Transformer experiments, uniform-in-depth guarantees, and general claims of practical speedup remain outside this project.

Can the training dynamics of intractable nonlinear neural networks be approximated by those of n tractable linear neural networks? Can this approach yield tractable predictions of the evolving neural tangent kernel (NTK) and other relevant observables?

Much of what we know about neural network training comes from limiting models or specially structured models whose parameter dynamics are solvable. Real more complex networks need not satisfy their assumptions, and numerical trajectories rarely explain why a plateau ends or a representation changes. We propose a middle ground: replace one intractable flow with several explicitly solvable models, then control how their local errors accumulate. If successful, this could yield a finite-time map of learning regimes: which modes grow, which directions rotate, and which layers reduce loss, without assuming a fixed NTK throughout training. If unsuccessful, it would show how quickly nonlinear feedback destroys local solvability and identify architectures or phases where simulation or different reduced descriptions remain necessary.

Deep linear networks admit exact solutions in restricted regimes, elucidating plateaus and ordered mode learning [1]. However, linear input–output maps do not imply linear or integrable parameter flows, and constant forcing generally invalidates their conservation-based reductions. The NTK gives a fixed-kernel description in suitable limits [2], while lazy-training theory characterizes when nonlinear paths stay close to linearized ones [3]. Other parameterizations allow substantial feature learning [4], so a single kernel at initialization can miss changing representations.

There are also methodological precedents. Iterative linearisation repeatedly refreshes tangent models and empirically investigates feature updates [5]; exponential Rosenbrock integrators repeatedly solve locally linearized ODEs and analyze their accuracy [6]. Restricted linear-attention models yield some tractable dynamics [7], but do not provide a general local approximation theory for nonlinear architectures. What remains open is a criterion for useful analytic tractability: can we choose a surrogate family whose corrected dynamics are explicitly solvable, whose validity lasts long enough to reveal mechanisms, and whose error bounds transfer to observables of the original nonlinear network? Existing results do not establish this combination across most architectures.

Consider a nonlinear neural network whose learning dynamics we first approximate by a gradient flow on a time interval [0,T], with loss L and evolving parameters z(t). Write the original flow as $\dot z(t)=F(z(t))=-\nabla L(z(t))$. Suppose these dynamics are intractable, meaning that simple ODE methods cannot summarize them. Partition [0,T] into n intervals [tₖ₋₁,tₖ]. On each interval, choose a linear network from an explicitly solvable family with surrogate loss $\tilde L_k$. At a reference point aₖ, add a constant correction to match the original flow’s velocity:

This correction matches the velocity at aₖ and corresponds to adding a linear term in the parameters to the surrogate loss. The corrected system must itself remain explicitly solvable. Solve it on the interval, then restart the next surrogate at the computed endpoint. Denote the local approximation error by εₖ and account for its propagation through subsequent intervals. We then seek a finite-time error bound for the stitched trajectory $\hat z(t)$:

Under appropriate assumptions, the error should tend to zero as the partition is refined. For a prescribed error tolerance ε, our goal is to estimate the number of intervals n required. For relevant examples, such as a simple transformer, we seek a construction in which both n and ε are relatively small.

We can track the original network’s NTK along the parameter path. Under smoothness assumptions, parameter error can bound NTK error. We can also analyze effective rank, eigenspace rotation, residual-aligned learning rates and changes in activation or attention patterns.

If the training Jacobian and key backpropagation interactions vary slowly, we expect a few solved intervals [tₖ₋₁, tₖ] to reproduce loss and NTK evolution even when the kernel changes substantially overall. Local mode spectra may identify periods of rapid residual decay, alignment, or stagnation.

We would reject this direction if tractability consistently required intervals as short as standard integration steps.

Three months would produce one theorem with explicit assumptions, a reproducible implementation, and a benchmark that includes failure cases. In month one, we would derive solvable families. In month two, we would prove error propagation bounds and develop an adaptive interval rule, with a separate bound for any splitting approximation. In month three, we would compare the NTK and other observables and test one restricted transformer attention example.

Self-supervised objectives (SimCLR [1], BYOL [2], SimSiam [3], Barlow Twins [4]) reach similar benchmark performance. Do they get there through the same gradient dynamics? In shared linear models, what does each objective learn, in what order, how fast, and why?

Suppose all objectives learn the same features in the same order, and differ only in how fast each feature is learned. Then choosing an SSL loss is mostly a choice of speed and stability. Results transfer across objectives, and hyperparameters can be mapped between them.

Suppose instead they differ. Then we get a dictionary linking each mechanism (negatives, decorrelation, or a predictor with stop-gradient) to concrete dynamical effects. It tells us which features are learned first, late or never, and how initialization scale, step size, batch size and noise change that.

SSL encoders are the perceptual front end of many agents and world models. Predicting what they learn from how they are trained is a step toward understanding and controlling these systems.

Learning mechanics gives exact, ODE-level accounts of supervised linear and diagonal networks: sigmoidal, mode-by-mode learning [5], conservation laws that come from symmetries [6], the effect of initialization scale and imbalance, rich versus lazy [7, 8, 9], saddle-to-saddle dynamics in the vanishing-initialization limit [10, 11], the effect of finite steps and SGD noise on what is learned [12, 13, 14].

For SSL, dynamical results come one objective at a time: Tian et al. [15] analyse BYOL and SimSiam and how the predictor aligns with the input correlations. Simon et al. [16] solve linearized Barlow Twins from small initialization and find stepwise learning of eigenmodes. Jing et al. [17] and Ziyin et al. [18] study dimensional collapse through stationary analysis rather than trajectories. Littwin et al. [19] show that deep linear self-distillation, as used in JEPA-style training, favours high-influence features. HaoChen et al. [20] characterise contrastive optimum spectrally, but not how training reaches it.

No work applies the full toolkit to several SSL objectives in one common model. That toolkit covers: exact ODE solutions and conservation laws, vanishing-initialization limits, finite-step maps, SDE and Fokker–Planck descriptions of noise.

Without such a comparison we cannot tell which dynamical differences are intrinsic to the objectives.

• 1.Common model. Gaussian data with spectrum $\Sigma$; augmentations as noise with covariance $\Sigma_\mathrm{aug}$; a two-layer linear encoder $W = W_2W_1$ and its diagonal variant $u\odot v$; a predictor head for BYOL and SimSiam [2, 3]. Write InfoNCE (in the large-batch limit), Barlow Twins and BYOL/SimSiam as functions of the end-to-end map, and derive their gradient-flow ODEs.
• 2.Find the symmetries and conservation laws [6]. Ask which ones the predictor, stop-gradient and EMA break; EMA makes the flow non-gradient.
• 3.When the covariances commute, reduce to mode-wise ODEs. Solve them exactly (logistic or Riccati type [5, 16]) or derive time-scale asymptotics, and identify the quantity that orders the modes.
• 4.Vanishing initialization: rescale time by $\log(1/\alpha)$ and derive the limiting staircase of jump times and plateaus [10, 11].
• 5.Rich versus lazy: study initialization imbalance and kernel evolution [7, 8, 9], and give a mirror-flow description of the implicit bias where one exists.
• 6.Finite steps: write discrete maps for the conserved quantities, as in [12].
• 7.Noise: model SGD as an SDE with multiplicative noise and write the Fokker–Planck equation for the mode amplitudes. Derive when a weak mode is killed (stochastic collapse [14]) or effectively re-initialized [13]. In high dimension, pass to a continuity PDE for the density of mode amplitudes.
• 8.Test the predicted orderings, time scales and noise thresholds on small MLP and CNN encoders (synthetic data, CIFAR-10), tracking per-mode alignment and effective rank.

Prediction: from small initialization, every objective learns augmentation-invariant modes stepwise, in an order set by a per-mode augmentation signal-to-noise ratio. The objectives differ in three ways:

• 1.their mobilities (sigmoidal versus exponential onset);
• 2.their conserved quantities;
• 3.the lengths of their plateaus.

We also predict that minibatch noise acts multiplicatively on weak modes. Dimensional collapse can then arise as a stochastic-collapse effect, with a threshold set by batch size and step size. It should be stronger for objectives without an explicit decorrelation term.

We would drop or reframe the question in either of two cases: Already in the linear model, all objectives reduce to the same ODEs up to a rescaling of time. Real networks show neither stepwise ordering nor a noise-dependent loss of weak modes.

Month 1: Set up the common model and derive the gradient-flow ODEs for common SSL models. Find their symmetries, conservation laws, reduce to mode-wise ODEs, and obtain exact or asymptotic solutions in the commuting case, checked by numerical integration.

Month 2: Derive the vanishing-initialization staircase and the rich-vs-lazy behavior for all 3 objectives, and the finite-step maps. Build the SDE model of minibatch noise and carry out the Fokker–Planck analysis of weak-mode collapse for one objective. Start experiments on synthetic data.

Month 3: Test the predicted orderings, time scales and noise thresholds on small encoders (synthetic data, CIFAR-10). Assemble the comparison table and write up.

Resources: An agent that would help us handle the theoretical derivations and computations. The experiments need a single GPU, roughly 100–300 GPU-hours.

Does a transformer trained on a star free/regular language, by next token prediction, does it have an algebraic structure beyond what prediction strictly requires, and if so, when does that structure emerge (before or after generalisation)?

The general idea; claims that models "learn maths" or "world models" dont really separate learning the underlying structure from learning just enough to do the task. Formal languages make the difference clear.

If monoid structure appears

Networks build algebraic structure beyond what is strictly necessary for token prediction

That gives interpretability a known target to search for, instead of reverse engineering structure after the fact.

If only states appear

Prediction models learn the minimal sufficient statistic and nothing else

That's a general check before any "model learned this thing" claim

If the algebraic class predicts the mechanism

Formal language theory might predict internals, not just behaviour.

In safety terms, it's a toy demonstration of predicting generalisation from internals before deployment

State tracking [1]: transformers trained on permutations represent maps between states rather than states. But permutations form a group, so "learned the algebra" and "learned the minimal statistic" can't be distinguished.

Algebraic interpretability [2, 3]: networks learn group composition via irreps. The extension to monoids works only where structure is locally invertible. Aperiodic monoids have trivial maximal subgroups, so the group-character methods used in [2, 3] have nothing to find.

Computational mechanics [4]: transformers linearly represent belief states. This is geometric, not algebraic.

Formal language theory [5, 6, 7]: establishes which languages transformers can express, and which they length generalise on (C-RASP). These results are constructive/behavioural, afaik none look inside trained networks.

Grokking [8]: structure before generalisation, but shown on single tasks whose structured solution was identified after, never on a task family with known competing representations.

• 1.Choose languages

For small star free languages, compute the monoid and measure size vs automaton's states, over the segments the model will actually see.

Keep languages where the gap is large (because this is the place where ‘learns the algebra’ vs ‘learns just enough to predict’ make different enough predictions to actually find.

Pick some matched pairs: a language transformers generalise on, for example a language expressible in C-RASP (a counting based formalism whose languages transformers provably length generalise on) and one that isn't expressible in this way.

• 2.Training

Small transformers, next token prediction on a fixed training set, with held out contexts and longer test sequences.

Sweep training set size and weight decay to find a regime where generalisation is delayed

• 3.Test what the final model represents

Monoid vs state, are chunks of input grouped by their monoid element, or only by their effect on the current state?

Counterfactual probes, ie, can you read out what a chunk would do from states the model isn't in? (Only a monoid representation should support this.)

Algebraic checks, do segments the monoid says are equal (like repeated letters that stop mattering) get identical representations?

The control is random groupings of chunks with the same class sizes as the monoid partition, nested within the same state classes, so that any extra explanations can't come from just having more labels

Test causality by intervening on the structure and check that predictions change as the algebra says they should.

Predictions

Intermediate layers encode chunks by monoid element, not just their effect on the current state.

Counterfactual probes read out what a segment does from states the model isn't in, above random grouping controls.

Intervening on this structure changes predictions as the algebra dictates.

Over training, monoid structure appears before the loss drops from the excluded data.

Final layers compress it down to states.

The generalising language is represented via counters, the other via shortcuts that break on longer inputs. (See Yang et Al, [7])

Would change my mind based on

Would drop the monoid question if, across several languages, monoid class never explains chunk representations better than current state does, and counterfactual probes stay at control level. That result means prediction trained transformers learn only the minimal statistic, not algebra.

Drop the counter hypothesis if the generalising language shows no linear features, or both languages are represented the same.

A pipeline: language selection, training, ground truth labels, etc

An answer to whether monoid structure is represented and causally used, on 2–3 languages.

Hopefully, emergence timing relative to generalisation.

One matched C-RASP / non-C-RASP contrast.

Needs not that much compute, on the order of a few hundred GPU-hours (rough Claude estimate).

Beyond three months but potentially doable depending on capabilities acceleration with new models in the loop: group components and a Hankel rank grokking sweep.

For quanta with Zipfian use frequencies $p_k \propto k^{-(1+\alpha)}$ placed on a random prerequisite DAG, where a model of capacity $n$ learns the downward-closed set of $n$ quanta with the largest total frequency, is the parameter-scaling exponent exactly $\alpha_N = \frac{\alpha}{(1+\delta)}$, where $\delta$ is defined by the prerequisite closure of the $m$ most frequent quanta having size ~ $m^{(1+\delta)}$? In particular, does this recover $\alpha_N = \alpha$ whenever prerequisites rarer than their dependents (“frequency inversions”) are asymptotically sparse?

The quantization model of Michaud et al. assumes: “QH2 (ordered acquisition). Networks learn quanta roughly in order of decreasing usefulness, so a given model has learned the first m quanta of a fixed sequence.” (from “Physics of Deep Learning - Part III Exercises: Neural scaling laws,” Iliad Intensive, October 2026; c.f., [1])

If our claim holds, the scaling exponent is set by two measurable properties of the data, not one: the frequency tail $\alpha$ and the prerequisite-closure growth $\delta$. That has three consequences:

It explains why common skills can emerge late: their ancestors are rare.

It turns emergence forecasting into a closure-cost calculation.

It suggests a concrete data intervention: upweighting rare prerequisites lowers δ and should steepen scaling.

It could also account for the gap between the gradient-cluster frequency exponent (about 0.24) and the measured $\alpha_N$ (about $0.08$), which $\delta \approx 2$ would produce.

If it fails, optimal learners skip expensive frequent skills, or $\delta$ is ill-defined. Then independence is not a harmless simplification, exponents cannot be read off frequency statistics, and power-law scaling under dependencies needs a different combinatorial invariant.

No existing result computes a scaling exponent for quanta with prerequisite structure.

The quanta model [1] derives $\alpha_N = \alpha$ by assuming independence: capacity goes to the $n$ most frequent quanta. This is valid only if the top-$n$ set is closed under prerequisites.

Compositional multitask sparse parity [2] shows empirically that prerequisites re-order learning, so frequency alone does not set acquisition order. It measures no exponent and proposes no invariant.

The Resource model [3] simulates learning on arbitrary dependency graphs and reproduces serial “Domino” acquisition. It solves only the forward problem on small hand-built graphs and gives no asymptotics for random DAGs with power-law frequencies.

Leap complexity [4] makes hierarchy govern SGD time - but for the Fourier support of a single target, not a frequency-weighted population of skills.

Hierarchical-grammar learning curves [5] combine power-law rule frequencies with hierarchy. For next-token prediction, they find the exponent decouples from the frequency law, which is evidence that structure matters. Their hierarchy is a generative tree rather than a prerequisite closure, and no closure-growth quantity appears.

Skill-graph frameworks [6, 7] use skill dependencies or co-occurrence to explain emergence and order curricula, but derive no scaling exponents.

The missing object is $\delta$: how prerequisite-closure size grows with frequency rank, together with whether greedy closure is asymptotically optimal.

The first step is to construct three random-DAG ensembles over quanta ranked by frequency, $p_k \propto k^{-(1+\alpha)}$:

(E1) Monotone. Each quantum draws d parents from more frequent quanta. This is the null case, $\delta = 0$.

(E2) Inversion-tunable. Parent ranks are drawn $\propto j^{-\theta}$ over all ranks, so $\theta$ controls how often a prerequisite is rarer than its dependent.

(E3) Layered. Frequency decays geometrically with depth.

For graphs of up to $10^7$ nodes I would compute the closure size $C(m)$ of the top $m$ quanta by reverse BFS, then fit $\delta$.

Second, I would compute the true optimal ideal, not just the greedy closure. The Lagrangian problem, maximizing $\sum_{k \in I} (p_k - \lambda)$ over ideals $I$, is a maximum-weight closure problem solvable by min-cut (Picard). Parametric min-cut (Gallo–Grigoriadis–Tarjan) gives the full nested family as $\lambda$ varies, which traces the concave majorant of captured mass against $n$. Comparing this to the greedy curve tests directly whether greedy closure is asymptotically optimal, and that result fixes the statement of the theorem.

Third, I would prove the result for E2. Exploring the ancestry of a quantum backwards is a branching process in rank. Its offspring tail determines $\delta$, and a criticality threshold beyond which closures span the graph. The goal is matching upper and lower bounds giving $\alpha_N = \frac{\alpha}{1+\delta}$.

Fourth, I would extend the result to data. A quantum is learnable only if it and all its ancestors appear $\tau$ times, which predicts $\alpha_D$.

Fifth, I would build compositional multitask sparse parity with E2-generated DAGs at fixed $\alpha$, vary $\theta$, train MLPs, and compare the measured $\alpha_N$ to $\frac{\alpha}{1+\delta}$.

Predictions:

E1 gives $\delta = 0$ and $\alpha_N = \alpha$.

In E2, $\delta$ grows continuously from $0$ as inversions become more frequent, up to a supercritical threshold. Beyond it, ancestry explodes, $\alpha_N \to 0$, and the loss plateaus.

Parametric min-cut matches the greedy exponent: skipping a frequent quantum forfeits mass that cannot be recovered polynomially.

In networks, the measured $\alpha_N$ tracks $\frac{\alpha}{1+\delta}$ within $\pm 0.05$ for $\delta \le 1$.

What would make me drop the question:

The optimal ideal beats greedy closure by a polynomial factor in natural ensembles. Then $\delta$ is the wrong invariant, and I would pivot to characterizing the min-cut hull directly.

No natural ensemble produces power-law closure growth; closures are either linear or explode. Then dependencies add no new exponent.

Networks learn dependent subtasks without their prerequisites (shortcut solutions). Then the DAG is not enforced, and the empirical arm fails.

Link to full proposal: https://docs.google.com/document/d/1AP0cUpMR8pxEtTvv4JMUlps2z9Zgu9Bhe0yFmsO3p88/edit?usp=sharing.

Month 1: ensemble generators, closure computation, and parametric min-cut. This is CPU-only and needs networkx and a max-flow library.

Month 2: branching-process proofs for E2, the criticality threshold, and the data exponent.

Month 3: compositional sparse-parity experiments, about 200 small-MLP runs (roughly 50 GPU-hours), and the write-up.

Deliverables: a theorem for E2, numerical phase diagrams for E1–E3, and a controlled empirical test.

Needs: a mentor fluent in random graphs and branching processes, plus modest GPU access.

If it overruns, drop first the transformer runs, then the data exponent. The theorem and the numerics are the core.

At the edge of stability, gradient descent holds the Hessian's top eigenvalue at 2/η and progresses along less sharp directions. Can each less sharp direction's curvature and remaining distance to its minimum predict long-run training speed?

Training dominates ML's cost, yet no mechanistic account explains how fast loss falls or why loss curves look as they do. Central flows [1] made the edge-of-stability regime predictable but leave "the global rate at which training loss decreases" open. A spectral answer would make it predictable: if Hessian eigenvalues break up into log-spaced clusters, each loss drop is one cluster being fit, and the spacing between clusters sets when drops come. It would also show how step size changes that schedule. Overtrained models are harder to fine-tune [2], and alignment relies on fine-tuning. If the current curvature says the loss won't drop again for a long time, stopping now gives up little improvement and keeps the model fine-tuneable.

Full-batch gradient descent typically runs at the edge of stability: the top Hessian eigenvalue rises to 2/η and hovers there while the loss keeps falling [3]. Central flows predict the time-averaged trajectory accurately [1], and Berthier [4] derives them by separating fast oscillation, self-stabilization, and a slow drift that lowers the loss. None of this work, including follow-ups [5, 6], asks how fast that slow drift goes.

Two findings say where to look. Progress happens along the less sharp directions: at the edge of stability, updates restricted to the top eigendirections stop lowering the loss, while updates restricted to the rest train normally [7]. And those directions are sloppy [8]: trained networks' Hessian eigenvalues spread over many decades [9], in clusters [10]. For linear models, eigenvalues plus each direction's distance to the minimum determine the loss curve [11]. We know of no such account at the edge of stability, where the spectrum changes during training.

Separately, overtrained models fine-tune worse [2], and models lose the ability to keep learning as they lose curvature directions [12], but these effects are measured after the fact, not forecast from the spectrum.

We measure each less sharp direction's curvature and remaining distance (how far the weights are from the valley's bottom along it), predict the loss curve, and test the prediction:

Toy models. Losses with log-spaced eigenvalue clusters below the top, chosen distances, and a cubic term so gradient descent settles at the edge of stability [13]. The prediction, $\mathrm{loss}(t) \approx \min + \sum \tfrac{1}{2} h d^2 \exp(-2\eta h t)$ for curvature h and distance d, should match gradient descent and the central flow.

Curve collapse. With the central-flows code [1], train its small networks (e.g., a vision transformer on a CIFAR-10 subset) at five step sizes. Each less sharp direction moves at a speed proportional to η × its curvature. So if those directions set the rate, loss plotted against η × steps should collapse onto one curve at long times. Whether it does is the first result.

Snapshot prediction. At checkpoints, estimate eigenvalues below the top with Lanczos (using only Hessian–vector products), started from the gradient. Since gradient = curvature × distance, this gives each direction's distance. Predict the remaining loss curve and compare with gradient descent and the central flow.

Diagnose failures. Do the less sharp eigenvalues rise toward 2/η, or does the oscillations' push toward flatter regions [1] act on the less sharp directions?

Stopping. Stop some runs where the snapshot predicts a long plateau and continue others. Fine-tune both on held-out classes and test whether the early-stopped models fine-tune better at little cost in loss.

Toy models: loss falls in stages, about one cluster per decade of training time, each ending after roughly 1/(η × cluster curvature) steps. The slow drift of [4] splits into one timescale per cluster.

Networks: the snapshot should hold for a while, then drift. We guess the less sharp eigenvalues creep toward 2/η, as top eigenvalues do [3], speeding those directions up. If so, curves at different step sizes won't fully collapse, and the mismatch measures how much edge-of-stability dynamics reshape the spectrum.

Stopping: runs stopped at a predicted long plateau should fine-tune better than runs trained past it, at small cost in loss.

What would change our minds:

If curves collapse and the snapshot holds throughout, the rate is a simple spectral quantity and the central flow matters only locally.

If early stopping doesn't help fine-tuning, fine-tuneability isn't tied to the loss curve's stages.

Month 1: toy models with designed spectra, confirming the snapshot prediction where the answer is known, plus curve-collapse tests across five step sizes on the central-flows paper's small networks. Month 2: snapshot predictions at checkpoints on two or three of those networks and, where the predictions drift, identifying which of the two causes in step 4 is responsible. Month 3: the stopping test (early-stopped vs. longer-trained models, fine-tuned on held-out classes) and a workshop paper, with code for measuring curvatures and remaining distances along training runs.

Do aggregate validation metrics in LLMs hide separate grokking events for distinct capabilities, events that small models trained on two algorithmic tasks can tell apart, including whether aligned and misaligned behaviors generalize through separate events in the same model?

Evidence of asynchronous, domain-specific grokking suggests that aggregate performance may obscure distinct generalization transitions. Training small models on paired algorithmic tasks, with separate evaluations and single-task controls, would provide a tractable setting to investigate this possibility. Extending these diagnostics to safety-relevant behaviors could help determine whether data sets that induce specific alignment and misalignment generalize at different training stages, informing when and how to intervene.

Grokking has primarily been studied in small networks trained on isolated algorithmic tasks. Its role in larger language models is less understood. Although recent work reports asynchronous, domain-specific grokking during LLM pretraining, it remains unclear whether aggregate validation metrics conceal multiple transitions associated with distinct learned features. Existing domain-level observations do not establish whether these transitions reflect separable mechanisms or interacting generalization processes. Controlled experiments combining multiple tasks could help resolve this ambiguity. Whether similar transitions govern the generalization of safety-relevant behaviors and whether they provide useful opportunities for intervention remains an open research direction. Li et al (2023) [1].

We will track how the losses on a small set of test inputs rise and fall together during training [3]. A closely related quantity has been used to find internal structure in small language models [4]. Neither has been tested on two learning events that overlap in time. Wang’s measure [2] has tracked only a single event in small networks.

The safety question has not been measured. A model further trained on a narrow set of harmful examples can abruptly become broadly misaligned [5]. Whether learning the narrow harmful behavior and becoming broadly misaligned happen as separate events is unknown.

Start with a simple prediction. Work out when an overall score hides a jump in one task: roughly, when the jump affects few test examples or overlaps with steady improvement elsewhere.

Start with two tasks. Train small transformers on two algorithmic tasks and test each separately. Compare against single-task models and against models that generalize gradually or only memorize. To move beyond toy settings, also mix both tasks into ordinary text; changing how often each appears moves their learning events closer together or further apart.

Look for separate learning events. Check whether each task improves at a different time while overall accuracy changes smoothly, and whether our prediction says correctly which jumps stay hidden.

Find better measurements. Without telling the method which task is which, track internal changes tied to each task. Use the 2RDM [3], which shows which test inputs change together, along with activation patterns or directions. Measure how close two events can be before each method stops separating them, and whether it notices each event before accuracy jumps.

Adapt Wang’s scaling method [2]. Measure how gradient cascades or responses to small activation perturbations grow with model size, track the resulting exponents during training, and compare them with each task’s improvement.

Check real language models. Apply the best measurements to OLMoE’s public training checkpoints [1] to see whether its domain-level jumps split into separate events.

Explore safety applications. In setups where narrow training produces broad misalignment [5], do these measurements separate learning narrowly vs. broadly.

Separate learning events: Different tasks may generalize at different times, while overall accuracy changes smoothly. Consistently synchronized improvements would weaken the masking hypothesis.

Distinct internal signals: Some activation patterns may track individual tasks. If they only reflect overall performance, they would offer little help in separating learning events.

Changes in scaling exponents: Exponents may change near each task’s generalization event. Similar changes in memorization-only or gradual-learning controls would weaken their value as grokking indicators.

Effects of intervention: Altering a task-linked internal signal may selectively delay or advance that task’s generalization. No selective effect would weaken a causal interpretation.

Relevance to safety: Narrowly harmful behavior and broader misalignment may emerge at different stages. If the measurements cannot distinguish them, the proposed safety application would need reconsideration.

Three months: a resolution test of how close two learning events can be and still be told apart, an open toolkit for the measurements, a check on OLMoE checkpoints, and a first misalignment test. If time runs short, drop the OLMoE check first, then the comparison with Wang’s method.

What training dynamics emerge when an LLM learns a Hidden Markov Model (HMM), and how does this compare to the idealised Hidden Markov Model given by probability theory? How do the Lazy and Rich regimes differ for unifilar and nonunifilar Markov processes?

A small transformer trained only by policy gradient to match a transducer in a partially observable environment will keep just a utility-relevant representation of its Bayesian belief in its residual stream, not the full belief geometry that next-token prediction produces.

Several safety strategies assume we can read an agent's world model: auditing its beliefs, predicting its behaviour off-distribution, and catching mismatch between what it believes and what it does. They fail quietly if we look for the wrong object.

If yes, RL-trained agents are blind to distinctions their reward never made relevant, and those blind spots are where they fail off-distribution. Belief probes would report partial failure even for a well-functioning agent; the right probe target is a different, utility-aware belief geometry. Reward design would shape what an agent can perceive, not just what it does: a double-edged sword.

If not, reward alone induces full world models in this architecture, and belief probes are the right audit tool for RL agents.

• [1]shows transformers trained on next-token prediction linearly represent the belief-state geometry of the generating process. [2] establishes that attention implements a constrained belief update that MLPs warp into the full geometry. Neither of these references has actions or utility. [3, 4, 5] are theory papers explaining how transducers give a general formalism for world models. All three name empirical work on trained networks as progress for the future work. [6] shows agents competent at multi-step goals must encode a world model, but only fully observed environments are covered. The closest result is [7]: in recurrent Q-learning, hidden states come to carry the belief over control-relevant variables, while information about irrelevant variables declines. We modify three things. Transformers need not carry a recursive state, unlike RNNs. Pure policy gradient removes value prediction from what shapes the representation. And small environments give exact ground truth in place of estimated mutual information.

• 1.Define environments and compute belief geometries. Each environment is a small partially observable transducer: finite actions $x$ and observations $y$, at least three hidden states $s$, and transition kernels

The generator would be non-unifilar, so beliefs never synchronize with the actual underlying state. The reward is $u(s_t)$ for a utility function $u$ on hidden states. A belief $b$ is a probability row vector over $S$. A future word $w$ is a sequence of (action, observation) pairs. $\tau(b, w)$ applies the updates in order. $W(b)$ is the set of words with positive probability under $b$. Before any training, we would compute three partitions of the belief geometry:

• •Predictive, $\sim_P$. $b \sim_P b'$ iff $b$ and $b'$ give every future observation sequence the same probability, under every action sequence. This is the equivalence used by the $\varepsilon$-transducer.
• •Value, $\sim_V$. $b \sim_V b'$ iff $W(b) = W(b')$ and $Q^*(\tau(b, w), x) = Q^*(\tau(b', w), x)$ for every action $x$ and every $w \in W(b)$. The empty word is included.
• •Action, $\sim_A$. $b \sim_A b'$ iff $W(b) = W(b')$ and $A^*(\tau(b, w)) = A^*(\tau(b', w))$ for every $w \in W(b)$.

We would keep only environments where all three differ.

• 2.Train agents. We would train a decoder-only transformer (about 4 layers, 64-dimensional residual stream, 4 heads, as in [1]) which reads "BOS $x_1\, y_1\, x_2\, y_2\, \ldots$" and outputs an action distribution; note the reward is not in the context. Episodes end with probability $1 - \gamma$ per step. A separate critic $V$ keeps the policy's representation shaped only by action choices. The objective would probably be something like:

Comparison regimes on the same environments: next-token prediction under a uniform random policy (control) and actor-critic with a shared trunk. We can use PPO instead of plain policy gradient if the variance is too high.

• 3.Probe. We can label every history with its exact belief, Q-values and quotient classes. Take residual-stream activations at each decision position, per layer. Choose history pairs with different beliefs but the same quotient class and test whether linear and small nonlinear probes separate them, on on-policy and random-policy histories. Controls: shuffled labels, position regressed out, an untrained network, near-optimal runs only.
• 4.Extension. Another natural question is how RL post-training on a pretrained next-token predictor model alters the belief-geometry it represents in the residual stream. So we could repeat the previous experiments with pre-trained next-token prediction models and check if there is a difference compared to ab initio RL-trained models.

Prediction. The transformer only learns what it needs to maximize utility. So each training regime would preserve a different geometry:

• •next-token prediction keeps the full belief geometry;
• •Q-learning or a shared-trunk actor-critic keeps the value-relevant quotient;
• •Policy gradient with a separate critic keeps only the action-relevant quotient.

Distinctions an objective does not require are absent, not merely nonlinear: nonlinear probes cannot decode them, even on random-policy histories.

What would change our mind. If near-optimal policy-gradient agents retain distinctions their objective does not require, decodable on random-policy histories, we drop the coarse-graining claim: reward alone induces full world models in this architecture.

In three months: the quotient definitions and a small benchmark of environments with known ground truth; a per-regime answer to which distinctions survive empirically in the residual stream, written up as a paper with released code.

Groups of aligned agents interacting and exchanging information can produce unexpected and misaligned collective behaviors. Can we characterize this emergence by employing a recently developed information-theoretic framework to study how information is shared and processed among different interacting agents?

As AI systems increasingly involve multiple interacting agents, understanding how individually aligned agents can produce collectively misaligned outcomes is becoming a central safety challenge. Recent research shows that interactions can generate collective biases absent in individual agents, with unexpected effects depending on group size [1]. Moreover, Anthropic demonstrated that organizations of aligned agents can achieve better performance while producing less aligned outcomes [2]; further emphasizing the risks emerging from interacting AI systems [3].

These findings highlight the need for a general framework to characterize and understand emergent collective behaviours. A theoretical and computational approach could both investigate underlying mechanisms in toy models and provide practical tools to measure interactions in multi-agent systems, ensuring a solid step towards safe multi-agent systems.

Recent contributions have highlighted both the risks emerging from multi-agent interactions and the development of mathematical frameworks to characterize agents and their behaviour.

In particular, two research directions are relevant:

Studies have shown that interactions among individually aligned agents can generate collective biases and misalignment, with nonlinear effects depending on group size [1,2]. More broadly, multi-agent risks have been classified into miscoordination, conflict, and collusion [3].

Computational mechanics has been employed to investigate fundamental limits of world modelling and agent evaluation [6], while information-theoretic approaches have provided rigorous definitions of plasticity and empowerment, characterizing how agents influence and are influenced by their environments [7].

However, in this context, we still lack a unified, theoretically grounded framework for investigating interactions among agents, which can be used both in toy models and in real-world scenarios. To address this gap, I propose extending existing information-theoretic frameworks [7,8] to investigate interactions among multiple agents and their environment. Such an approach could provide a mathematical foundation applicable across a broad range of settings, bridging the gap between theoretical models from computational mechanics and real-world multi-agent systems involving LLMs.

The first step consists of constructing the theoretical framework. I would start from the information-theoretic formalism recently developed by [7], where the empowerment and plasticity of an agent are defined with respect to its environment. This formalism is based on sums of conditional mutual information terms, allowing the application of Partial Information Decomposition (PID) [9].

First, I would combine PID with Abel's framework to define different information-sharing relationships, introducing synergistic (available only when two agents are considered together), redundant (shared equally by the two agents), and unique (characterizing one agent specifically) empowerment and plasticity. I would then investigate the mathematical properties of these measures, establishing their bounds and potential relationships.

Second, I would start with simple toy models and interaction games to illustrate the interpretation of these measures and benchmark them against known interaction structures.

Third, I would extend the analysis to more complex scenarios involving interacting LLM agents, including coordination games already investigated in the literature [1]. The goal would be to understand how results obtained in simple models translate to more complex systems and whether these measures can help explain mechanisms underlying emergent collective biases.

Finally, I would apply the framework to groups of interacting agents exhibiting collective misalignment, investigating whether these measures can track the emergence of unexpected behaviours and help reveal their underlying mechanisms.

As a whole, this project offers a general framework to characterize, define, and measure the emergence of unexpected collective properties from a group of individual agents.

First, I expect to develop a theoretically well-defined framework extending Abel et al. [7]. In particular, their work establishes a tight upper bound on the sum of empowerment and plasticity. I aim to extend these results by defining synergistic, redundant, and unique components of empowerment and plasticity, and investigating their mathematical properties and bounds.

Second, I expect simple models to reveal synergistic empowerment when groups of agents influence their environment only through their joint actions. For instance, an environmental response determined by a specific configuration of agents' outputs should exhibit synergistic contributions.

Finally, in LLM-based systems, I expect these measures to characterize coordination and track the emergence of collective biases and misalignment.

I would reconsider if our formalisms failed to capture group interactions in toy models or offered no insight beyond simpler information-theoretic measures. In this case, we would clarify where the formalism is useful and where it is not.

Within three months, I aim to produce a research paper connecting different levels of analysis: a mathematically well-defined framework, its validation on toy models, and its application to more complex multi-agent scenarios.

The main contribution would be a set of theoretical and computational measures to characterize collective behaviours that cannot be explained by individual agents alone.

Since the project does not require training large models, but only running existing models in different settings, I expect computational costs to remain limited (below $6,000).

If necessary, I would prioritize theoretical results and toy-model validation, postponing more complex LLM experiments.