Is there a characteristic geometric structure tha…

{
  "query": "grokking representation geometry Fourier structure modular arithmetic generalization intrinsic dimension causal intervention",
  "categories": [
    "cs.LG"
  ],
  "max_results": 10
}

{
  "title": "Grokking geometry: calibrated, causal test of representation-manifold formation",
  "research_question": "As a network groks, does its representation adopt a characteristic structure detectable without presupposing the answer, and is that structure causally tied to generalization rather than merely correlated with it?",
  "method": "Phase 0 calibrates angular, topological, dimensional, and reconstruction probes on synthetic circle, torus, sphere, line, non-abelian Cayley-graph, and random high-dimensional point clouds across noise and sample size. Phase 1 trains a standard one-layer transformer and a two-layer MLP on (a+b) mod 97 with AdamW and dense checkpoints; unit-normalized pairwise-angle distributions and persistent homology on cosine distance are primary, with participation ratio, stable rank, Fourier concentration, Procrustes/CKA drift, and lead/lag as secondary. Render PCA-evolution movies only for runs that grok. Phase 2 compares mod-add, mod-mul, a 2D grid task, S5 composition, and random relabeling. Phase 3 intervenes with correct versus incorrect architectural invariances, frozen correct/wrong/random manifolds, and low-dimensional bottlenecks. Phase 4 checks breadth on larger-but-fast models.",
  "controls": [
    "synthetic known-topology calibration before trained-model interpretation",
    "line point cloud: low-dimensional but no H1 cycle",
    "random high-dimensional point cloud: probe hallucination control",
    "random-relabeling task: memorization without generalizable structure",
    "correct versus wrong architectural invariance",
    "correct ring versus wrong low-dimensional and random frozen manifolds",
    "dimension bottleneck without a prescribed shape",
    "angle-based versus magnitude-based descriptors on identical checkpoints"
  ],
  "evaluation": "Per-checkpoint train/test accuracy aligned with angular-distribution distance, H1/H2 persistence, participation ratio, stable rank, Fourier concentration, and representation drift. Treat lead/lag as correlational only. Causal conclusions come from whether correct versus incorrect invariance or geometry changes whether and when grokking occurs. Compare task signatures against Phase-0 calibrations.",
  "experiments": [
    "Phase 0 known-manifold probe calibration",
    "Phase 1 mod-add descriptive trajectory on transformer and MLP with five seeds",
    "Phase 2 task-structure generality and random-relabeling negative control",
    "Phase 3a correct versus incorrect architectural invariance",
    "Phase 3b correct/wrong/frozen manifold and dimension-bottleneck interventions",
    "Phase 4 breadth and scaling battery"
  ],
  "deliverables": [
    "calibration table",
    "aligned geometry and accuracy curves",
    "PCA-evolution movies for grokked runs only",
    "cross-task geometry table",
    "causal intervention table",
    "metrics CSVs and written verdict"
  ],
  "compute": {
    "gpu_type": "A100-80GB",
    "num_gpus": 1
  }
}

{
  "proposal_id": "P-g4c1"
}

{
  "run_id": "R-g91e",
  "section": "summary",
  "detail": "completed phases, scientific failures, and unresolved causal gaps only"
}

{
  "run_id": "R-g91e",
  "instruction": "After completing the S5 rerun, add Phase 3c as an additive causal test. Keep transformer and MLP architectures and embeddings fully learnable on mod-add. Add a differentiable auxiliary loss that softly pushes the centered, unit-normalized embedding pairwise-cosine Gram toward the Phase-0 calibrated evenly spaced ring. Sweep coefficient 0, 0.5, and 2.0. Include a magnitude-only pressure with no angular content and a scrambled-ring target with identical topology but wrong token-to-angle correspondence. Measure ring distance, participation ratio, H1, grok onset, and final test accuracy. Do not replace or freeze the representation; this test is about pressure, not constraint."
}

{
  "run_id": "R-g91e",
  "section": "metrics",
  "detail": "final cross-task geometry, restrictive interventions, and additive Phase 3c condition means"
}

{
  "run_id": "R-g91e",
  "section": "controls",
  "detail": "verify which Phase 3c claims survive the controls and identify limitations"
}

{
  "title": "Soft angular ring pressure advances grokking on modular addition, but the initial evidence supports only a sign claim",
  "statement": "With embeddings left learnable, auxiliary pressure toward the calibrated ring moved grokking earlier for both a transformer and an MLP; magnitude-only pressure did not. The result supports angular structure as causally relevant, but not yet a precise magnitude or graded dose response.",
  "evidence": "Two seeds per condition on mod-add. Correct angular pressure: transformer -23%/-33% and MLP -32%/-33% at coefficients 0.5/2.0, final test 1.0. Magnitude-only delayed transformer +44% and left MLP at final test 0.14. Scrambled ring: transformer -26%, MLP -5%.",
  "caveats": "No per-seed error bars, one task, and only two active coefficient points. The MLP response is flat across the two doses. Frozen-manifold failures show that rigidly imposing a single-frequency ring is not equivalent to softly encouraging angular structure."
}

{
  "query": "soft angular ring pressure grokking sign claim limitations",
  "max_results": 5
}

{
  "title": "Phase 3c hardening: is angular-pressure acceleration real, graded, and cross-task?",
  "research_question": "Does the initial sign of earlier grokking under soft angular ring pressure survive proper seeding; how large is it; is it a graded coefficient response or a threshold; and does it replicate on modular multiplication?",
  "method": "Reuse the exact additive loss from the prior run. Train a one-layer transformer and a two-layer MLP on (a+b) mod 97 and (a*b) mod 97 with AdamW, weight decay 1.0, learning rate 1e-3, d=128, training fraction 0.3, and a 30000-step horizon. For addition use the evenly spaced ring Gram target. For multiplication use discrete-log order on the nonzero residues with T[i,j]=cos(2*pi*(dlog(i)-dlog(j))/(p-1)). Use ten shared seeds per condition. Define grok step as first test accuracy at least 0.95, censored at the horizon.",
  "controls": [
    "coefficient 0 baseline",
    "magnitude-only pressure at matched coefficients",
    "scrambled-ring target at matched coefficients",
    "final test accuracy guardrail",
    "grokked-only analysis to test censoring sensitivity"
  ],
  "evaluation": "Correct-angular coefficient grid {0,0.25,0.5,1.0,2.0,4.0}. Report per-condition mean, standard deviation, bootstrap 95% CI, Mann-Whitney U, paired Wilcoxon, rank-biserial effect size, and final test accuracy. Compute Spearman over the full grid and separately within active doses so the zero-to-positive cliff cannot masquerade as a graded response.",
  "experiments": [
    "mod-add coefficient grid on transformer and MLP",
    "mod-add magnitude and scrambled controls",
    "mod-mul discrete-log-ring coefficient grid on transformer and MLP",
    "mod-mul matched controls",
    "full-grid versus active-dose trend analysis and censoring-robust analysis"
  ],
  "deliverables": [
    "dose curves with uncertainty",
    "condition statistics table",
    "specificity-control figure",
    "one-paragraph verdict that explicitly chooses graded, saturating, null, or harmful"
  ],
  "compute": {
    "gpu_type": "A100-80GB",
    "num_gpus": 1
  }
}

{
  "proposal_id": "P-h7d2"
}

{
  "run_id": "R-h12a",
  "reason": "No work started. Reallocate the identical small-model batched study to the cheaper adequate A40; preserve all conditions, seeds, endpoints, and controls."
}

{
  "proposal_id": "P-h7d2"
}

{
  "run_id": "R-h84f",
  "section": "metrics",
  "detail": "full coefficient grid, uncertainty, full-grid and active-dose tests, and censoring-robust comparison"
}

{
  "run_id": "R-h84f",
  "section": "controls",
  "detail": "magnitude-only and scrambled-ring controls at matched coefficients, including grok reliability, terminal accuracy, and geometry"
}

{
  "title": "Soft correct-angular pressure robustly accelerates grokking as a saturating threshold",
  "statement": "Across modular addition and multiplication, soft pressure toward the task-correct ring geometry brings grokking substantially earlier in both a transformer and an MLP while preserving final accuracy. The effect saturates at the smallest tested positive coefficient rather than increasing monotonically with dose.",
  "evidence": "Ten shared seeds per condition. At coefficient 2: add transformer 21240 to 5720 (-73%), add MLP 13590 to 8930 (-34%), mul transformer 21530 to 5510 (-74%), mul MLP 11520 to 7110 (-38%); all Mann-Whitney p<0.001, paired Wilcoxon p=0.002, final test 1.0. Active-dose Spearman is nonsignificant: transformer -0.17/-0.23, MLP +0.21/+0.21. Magnitude-only delays transformers and yields 0/10 MLP groks; scrambled rings help transformers but not MLPs.",
  "caveats": "The result is established for two modular tasks and two small architectures. It does not establish the same intervention for non-cyclic task geometries. Transformer baselines include censored seeds, but grokked-only shifts remain -69% to -72%; MLP shifts remain -34% to -38%."
}

{
  "title": "Phase 2 hardening: is task-specific grokking geometry real beyond seed and dimension noise?",
  "research_question": "Do the different geometry signatures previously observed across modular addition, modular multiplication, a 2D grid, S5 composition, and random relabeling survive proper seeding and matched embedding dimension?",
  "method": "Reuse the calibrated geometry probes and task code. Run ten shared seeds per task. At d=128 run mod-add, mod-mul, grid2d, and random relabeling. At d=256 run S5, mod-add, and grid2d so S5 is compared against cyclic tasks at matched dimension. For grokked checkpoints measure raw and dimension-normalized participation ratio, stable rank, H1, H2, natural-index Fourier concentration, discrete-log Fourier concentration for multiplication, distance to ring, and angular entropy. For random relabeling report end-of-training descriptors and grok reliability.",
  "controls": [
    "matched d=256 add and grid runs for the S5 comparison",
    "dimension-normalized participation ratio; raw PR secondary",
    "random-relabeling negative control",
    "natural-index versus discrete-log-index Fourier for multiplication",
    "shared seed set across tasks"
  ],
  "evaluation": "Report mean, standard deviation, and 95% CI per task. Use Kruskal-Wallis across tasks followed by pre-registered directional Mann-Whitney contrasts with rank-biserial effects: grid H2 greater than add and mul; add natural-index Fourier greater than mul natural-index Fourier while mul discrete-log Fourier matches add; S5 lower circularity and higher normalized effective dimension than cyclic tasks at d=256; random relabeling does not grok. Report between-task versus within-task variance for each descriptor.",
  "experiments": [
    "d=128 core task battery with ten seeds each",
    "d=256 S5 plus matched add and grid controls",
    "pre-registered topology, indexing, circularity, and variance contrasts"
  ],
  "deliverables": [
    "per-task descriptor table with uncertainty",
    "grok-reliability table",
    "matched-dimension panel",
    "between-versus-within variance analysis",
    "verdict separating supported signatures from seed noise"
  ],
  "compute": {
    "gpu_type": "A100-80GB",
    "num_gpus": 1
  }
}