Stepwise preference

One step, scored, with the correction a researcher would make.

1/9

File metadata

_format:“option-a-stepwise-critique”

_note:“One element per scored step. Element shape is identical to rlhf-data-example.json (traj_id, step_id, category, task, history, current_step, after_step_observation, judge, score, correction) so an existing loader works unchanged. Ships as a JSON array here for…”

traj_id: 7e2f633e-4a27-4b9b-ad3b-7710590618d8
step_id: 3
category: research
task: grokking-representation-geometry
success: null
gt: null
score: 2

history

user

Grokking is a phenomenon in neural network training where a model first overfits the training data, then—often long after achieving near-perfect training accuracy—suddenly begins to generalize well to held-out data. This delayed generalization has been observed in algorithmic tasks and beyond, but its underlying mechanism remains poorly understood. Understanding what internal changes accompany or cause this delayed transition, and whether it can be predicted or accelerated, is important for improving training efficiency and for understanding how neural networks acquire generalizable structure. Research question I'm investigating: Is there a characteristic geometric structure that the representation space adopts as the model begins to generalize?

assistant

reasonGround the question in the grokking / representation-geometry literature before designing anything.

actionsearch_arxiv(query="grokking representation geometry Fourier structure modular arithmetic generalization", categories=["cs.LG"], max_results=8, sort="relevance")

observation

8 results. Nanda et al. progress measures; Liu et al. *Effective Theory of Representation Learning* (2205.10343); Gromov (2301.02679) derives the circular solution analytically; Kazanskii (2607.11666) uses representation geometry as a control signal. Established: grokked modular addition uses periodic/circular embeddings in a Fourier basis.

assistant

reasonThe known answer for cyclic tasks is a circle in Fourier space. The open parts are (1) detecting the structure without presupposing it, (2) whether the geometric signal leads the accuracy jump, (3) whether a different task adopts a different geometry. Check the GPU pool and draft a proposal.

actionlist_available_tier1_gpus()

observation

A100, A100-80GB, H100, A40 available.

current_step

assistant

reasonPropose the full study: track assumption-free geometric descriptors densely through the transition, test whether they lead test accuracy, and compare characteristic geometry across tasks. Hypothesis: generalization = collapse onto the manifold encoding the task's natural symmetry.

actionpropose_tier1(title="Grokking as manifold formation: assumption-free geometry of the generalization transition", gpu_type="A100", description="## Research question\\n\\nAs a network groks, does its representation space adopt a **characteristic geometric structure**, and can we detect that structure *without presupposing what it is*? The core hypothesis: **generalization = the representation collapsing onto the low-dimensional manifold that encodes the task's natural symmetry** (a circle/torus for cyclic modular arithmetic), and the geometric signature **precedes or coincides with** the test-accuracy jump, making it a predictive progress measure.\\n\\n## Setup\\n- Tasks: (a+b) mod 97, (a·b) mod 97, S5 permutation composition, random-relabeling control.\\n- Models: 1-layer transformer (Nanda config: d_model 128, 4 heads) and a 2-layer MLP.\\n- Dense checkpointing through the transition window.\\n\\n## Geometry / progress measures\\n- **Effective dimensionality:** participation ratio / stable rank of the representation covariance — expect a sharp *collapse* at grokking.\\n- **Fourier concentration:** DFT of embeddings; fraction of power in top-k frequencies + Gini sparsity.\\n- **Assumption-free periodicity:** persistent homology (H1) of the embedding point cloud.\\n- **Circular reconstruction quality.**\\n- **Lead/lag analysis:** cross-correlate each geometric signal against the test-accuracy curve to quantify whether geometry **predicts** the jump.\\n\\n## Experiments\\n1. Baseline grokking + geometry (mod-add, transformer, 5 seeds).\\n2. Architecture invariance (2-layer MLP).\\n3. Weight-decay sweep.\\n4. Operation ablation (mod-mul).\\n5. Symmetry-generality test (S5).\\n6. Memorization control (random relabeling).")

after_step_observation

observation

Proposal 4e2f7401-5930-4f79-b267-72cac01a18b0 created; awaiting researcher approval.

Researcher returned it for revision with eight inline plan comments. The first and longest:
- - [plan] "the representation collapsing onto the low-dimensional manifold that encodes the task's natural symmetry**" → how can you know what the correct low-dimensional manifold it is? this may accidentally be a circular reasoning argument: collapsing does not imply correct structure. And there may be multiple "correct" representation structures that can lead to reasonable performance. it's not immediately clear what the representation shape "should be" beforehand. We should dis-entangle this somehow. Maybe we should first test on a dataset where we know the structure (e.g. ring for the Fourier example)—although I'm not sure if this is the best way either.

and, free-standing at the end:
- "I guess one thing we could test initially is whether a significant representation collapse corresponds with a performance increase. We would have to have some metric of change in representation structure though and there are still the causal issues as mentioned in previous comments."

One section was praised: "I like this idea of not assuming the representation structure beforehand."

judge

The plan advertises itself as assumption-free and then writes the answer into the hypothesis. "Generalization = the representation collapsing onto the low-dimensional manifold that encodes the task's natural symmetry" presupposes both that there is a single correct manifold and that the researcher knows which one it is. 1. Circular reasoning: every descriptor in the plan (participation ratio, stable rank, Fourier concentration) measures *collapse*. If collapse is what you measure and collapse is what you hypothesize, a positive result is guaranteed and uninformative. Collapse does not imply correct structure — a 1-D line is maximally collapsed and encodes nothing. 2. No probe calibration: the topological probes are pointed at real training runs before anyone has checked what they read on a point cloud whose topology is known. An H1 number from an uncalibrated probe cannot be interpreted. 3. The researcher's own suggested fix — "first test on a dataset where we know the structure" — is the standard instrument-calibration move, and it is missing. The revision added a Phase 0 that ran the probes on synthetic circle / torus / sphere / Cayley-graph / random clouds. It immediately paid: the calibration found that a *line* has the lowest participation ratio of any manifold yet no H1 cycle, which is precisely the collapse-vs-correct-structure dissociation the researcher flagged. Without it the whole geometric story would have been unfalsifiable. {"Score":2}

correction

error_type:“presupposed_answer_uncalibrated_instrument”

first_problematic_step_id:3

critique:“- [plan] "the representation collapsing onto the low-dimensional manifold that encodes the task's natural symmetry**" → how can you know what the correct low-dimensional manifold it is? this may accidentally be a circular reasoning argument: collapsing does no…”

preferred_insertion_point:3

research_principle:“If your hypothesis names the answer, your measurement will find it. Calibrate the instrument on ground truth you already know before pointing it at the phenomenon, and add the intervention that would dissociate the thing you measure from the thing you claim.”