Long-Horizon Offline GCRL Value Errors

{
  "query": "offline goal-conditioned reinforcement learning long-horizon value bootstrapping stitching quasimetric architecture",
  "categories": [
    "cs.LG",
    "cs.AI",
    "cs.RO"
  ],
  "date_from": "2020-01-01",
  "max_results": 10
}

{
  "title": "Minimal causal screen of GCRL value failure",
  "research_question": "On the smallest deterministic offline GCRL benchmark that separates direct coverage from fragment composition, do inaccurate values arise from target/support failure, architecture, or their interaction? Geometry is a diagnostic control, not the main question.",
  "method": "Construct deterministic chain and branching graphs with exact DP values and BFS verification, at most 32 states and 8 selected goals. Build direct-trajectory and short-overlapping-fragment datasets with explicit boundaries. Compare tabular lookup, a small shared MLP, and an asymmetric quasimetric geometry model. Compare realistic learner-bootstrap TD1/TD4 and observed-return MC; retain oracle TD/DP supervision only as explicitly labeled capacity controls.",
  "controls": [
    "DP and shortest-path distances are evaluation-only in realistic learner conditions",
    "observed-return MC uses only recorded episode returns and never fabricated composed returns",
    "tabular lookup removes neural approximation",
    "matched plain-head CRL pretraining is diagnostic only",
    "finite-horizon versus infinite-horizon DP check"
  ],
  "evaluation": "True MSE/MAE against exact DP, Bellman residual, calibration, ranking, policy success, distance curves, and errors stratified by direct evidence, composition support, and unsupported pairs. Audit target construction and report any oracle use by condition.",
  "experiments": [
    "smoke and leakage audit",
    "tiny architecture x objective x data screen",
    "three-seed confirmation only for key interactions",
    "CRL matched representation ablation"
  ],
  "deliverables": [
    "raw JSON/CSV",
    "target-construction audit",
    "support-stratified table"
  ],
  "compute": {
    "gpu_type": "A10",
    "num_gpus": 1,
    "estimated_hours": 1
  }
}

{
  "proposal_id": "P-a103"
}

{
  "run_id": "R-b214",
  "section": "metrics"
}

{
  "run_id": "R-b214",
  "section": "controls",
  "detail": "ground-truth definition, target-source audit, MC implementation, boundary handling, and CRL matching"
}

{
  "title": "Initial GCRL screen is informative only after separating oracle controls from realistic objectives",
  "statement": "Learner-bootstrap TD can have low Bellman residual and high exact-DP value error, and geometry can reduce interpolation error without removing target error. The nominal MC condition cannot diagnose stitching because it used exact graph-distance targets; oracle TD1/TD4 are capacity/target controls, not realistic offline objectives.",
  "evidence": "On branch32 stitching, plain MLP oracle TD1/TD4/learner TD1 MSE was 0.043/0.266/0.375 with learner residual 0.029; geometry was 0.031/0.239/0.148 with learner residual 0.014. Target audit found the MC alias used exact graph distance.",
  "caveats": "The first run's boundary handling and CRL matching were insufficient for causal claims; finite-horizon truncation was already negligible by H=16."
}

{
  "title": "Leakage-free realistic offline GCRL stitching diagnosis",
  "research_question": "Can realistic offline objectives discover long-range goal values from short overlapping fragments, and which residual errors come from recursive bootstrapping, backup length, support, or representation?",
  "method": "Correct branch32 with explicit episode_id and step_in_episode fields. Give reward 1 only when the recorded trajectory reaches its labelled goal. Compare learner_td1, valid within-boundary learner_td4, and observed-return MC using identical data and update budgets. Run plain MLP, asymmetric geometry, and tabular lookup; exact DP is evaluation-only and oracle controls are separately named.",
  "controls": [
    "source-level leakage audit rejects V_star, distance, or DP calls in learner targets",
    "hand-constructed target tests for TD1, TD4, MC, terminals, and fragment boundaries",
    "direct-data MLP control before stitching matrix",
    "target coverage, nonzero fraction, bias, and variance computed post hoc only"
  ],
  "evaluation": "Exact-DP MSE/MAE, Bellman residual, ranking, calibration, policy success, distance curves, and support strata, with raw per-seed target diagnostics.",
  "experiments": [
    "smoke and leakage tests",
    "three-seed direct MLP control",
    "three-seed stitching matrix for MLP, geometry, and tabular"
  ],
  "deliverables": [
    "raw target audit",
    "per-seed and aggregate tables",
    "explicit warning for any target equivalence"
  ],
  "compute": {
    "gpu_type": "L40S",
    "num_gpus": 1,
    "estimated_hours": 1
  }
}

{
  "proposal_id": "P-c325"
}

{
  "run_id": "R-d436",
  "section": "summary",
  "detail": "first completed scientific checkpoint"
}

{
  "run_id": "R-d436",
  "instruction": "The completed direct MLP condition is a pipeline control, not the answer. Preserve the validated leakage and boundary implementation, and prioritize the missing branch32 stitching matrix: MLP, geometry, and tabular under learner_td1, learner_td4, and observed_mc with the same three seeds, training budget, and raw target diagnostics. Do not substitute more direct-data runs."
}

{
  "run_id": "R-d436",
  "section": "metrics",
  "detail": "completed three-seed stitching matrix and target-equivalence diagnostics"
}

{
  "title": "Leakage-free short-horizon stitching shows geometry helps, but TD4-versus-MC is unresolved",
  "statement": "On branch32 stitching, geometry reduces value error relative to a plain MLP and tabular lookup under the same learner TD1 objective, while low Bellman residual still coexists with high true error. TD4 and observed MC cannot be compared because they produced exactly identical metrics in every condition.",
  "evidence": "Three seeds and 27 completed runs: TD1 MSE plain MLP 0.0658+/-0.0041, geometry 0.0434+/-0.0017, tabular 0.2690+/-0.0030. Geometry TD1 stitchable/unsupported MSE was 0.007/0.073. Tabular TD1 Bellman residual was 0.0002 despite MSE 0.2690.",
  "caveats": "Branch32 is not genuinely long-horizon; TD4 and MC were exactly equal and MC is excluded until target construction is audited."
}

{
  "title": "Long-distance leakage-free GCRL value scaling",
  "research_question": "Do learner-bootstrap goal-conditioned values fail as maximum shortest-path distance and required fragment composition depth grow, and do TD1, TD4, geometry, and tabular controls locate the cause?",
  "method": "Build deterministic directed no-shortcut chain17/33/65/129 environments with maximum shortest-path distances 16/32/64/128. Use sparse terminal rewards, fixed four-step stitching fragments, 200 episodes, 150 epochs, and learner TD1/TD4 only. Screen the MLP at every scale; compare geometry and tabular at distance 128.",
  "controls": [
    "MC excluded until its exact TD4 equivalence is audited",
    "exact DP/BFS evaluation only",
    "fixed fragment length and approximately fixed data budget",
    "no shortcut edges",
    "target coverage and Bellman residual reported with true error",
    "policy success demoted on the directed chain"
  ],
  "evaluation": "MSE/MAE versus shortest-path distance and fragment count, support-stratified error, target coverage, ranking, calibration, Bellman residual, and endpoint architecture comparison.",
  "experiments": [
    "smoke, leakage, no-shortcut, reward, and n-step boundary tests",
    "MLP TD1/TD4 distance screen",
    "largest-scale geometry and tabular controls",
    "endpoint seed confirmation if metrics remain interpretable"
  ],
  "deliverables": [
    "distance-scaling table",
    "raw JSON",
    "mask definitions and counts"
  ],
  "compute": {
    "gpu_type": "A100",
    "num_gpus": 1,
    "estimated_hours": 1
  }
}

{
  "proposal_id": "P-e547"
}

{
  "run_id": "R-f658",
  "section": "metrics",
  "detail": "completed distance screen, endpoint architectures, coverage, residuals, and current evaluation strata"
}

{
  "run_id": "R-f658",
  "section": "controls",
  "detail": "exact definitions and counts for direct, stitchable, unsupported, selected goals, rewards, and DP usage"
}

{
  "run_id": "R-f658",
  "reason": "The completed distance screen establishes neural long-horizon failure, but the legacy direct/stitchable/unsupported masks are semantically invalid for the tabular and mechanism comparison. Additional seeds would tighten a misleading statistic. Retain completed predictions if available for corrected-mask evaluation."
}

{
  "title": "Long-chain GCRL exposes low-coverage value failure, while legacy support masks are invalid",
  "statement": "MLP and geometry values become severely inaccurate on no-shortcut long chains while Bellman residuals remain near zero, implicating sparse target coverage and local self-consistency. The legacy aggregate and direct/stitchable/unsupported metrics cannot support claims about tabular composition because their masks are not defined by observed evidence and are dominated by small discounted targets.",
  "evidence": "MLP TD1/TD4 MSE at max distance 32 was 0.253/0.251, at 64 was 0.321/0.541, and at 128 was 0.543/0.474; distance-128 stitching MAE was 0.703/0.690 with target coverage 0.021 and near-zero residual. Geometry endpoint MSE was 0.544/0.581. Tabular overall MSE 0.027 contradicted negative rank correlation -0.095 and direct-pair MSE 0.2583.",
  "caveats": "The distance curve is valid evidence of neural failure; the mechanism decomposition and tabular comparison require corrected masks and positive-value metrics."
}

{
  "title": "Auditable reevaluation of long-horizon GCRL values",
  "research_question": "Does the long-chain conclusion survive evidence-defined, positive-value, distance-stratified evaluation, and does tabular lookup fail genuine fragment composition as expected?",
  "method": "Add explicit fragment records and construct observed-direct, complete-trajectory, certified composition-required, and unsupported-reachable masks from contiguous recorded endpoints. Add positive-value thresholds, per-distance metrics, normalized error, scatter summaries, same-start ranking, and top-k recall. Reevaluate saved predictions; if predictions/checkpoints are absent, retrain the matching tiny matrix in the same run.",
  "controls": [
    "hand-constructed positive and negative composition-certificate tests",
    "mask counts and mean true value reported before errors",
    "DP used only for evaluation values and mask reachability, never targets",
    "legacy metrics explicitly labeled non-causal",
    "artifact round-trip if retraining is required"
  ],
  "evaluation": "MAE, MSE, normalized MSE, relative error, calibration, and ranking for every evidence mask, positive-value subset, and distance.",
  "experiments": [
    "implement and unit-test masks",
    "reevaluate historical chain17/33/65/129 predictions",
    "fallback: retrain MLP TD1/TD4 at all scales and geometry/tabular at distance 128"
  ],
  "deliverables": [
    "auditable mask counts",
    "corrected tables",
    "full prediction/checkpoint artifacts for future reevaluation"
  ],
  "compute": {
    "gpu_type": "L40S",
    "num_gpus": 1,
    "estimated_hours": 1
  }
}

{
  "proposal_id": "P-g769"
}

{
  "run_id": "R-h870",
  "section": "summary"
}

{
  "query": "this investigation long-chain GCRL leakage-free TD1 TD4 mask corrections MC exclusion",
  "max_results": 5
}

{
  "title": "Retrain and reevaluate long-horizon GCRL with auditable artifacts",
  "research_question": "Under corrected evidence masks, how do learner TD1 and TD4 estimate genuinely composition-required values across distances, and do geometry or tabular controls change the diagnosis at the longest endpoint?",
  "method": "Add chain17/33/65/129 aliases to the corrected leakage-free implementation. Retrain learner TD1/TD4, save full V_pred/V_star/distance matrices, checkpoint/config, exact transitions, fragment records, and target indices, then evaluate observed-direct, complete, certified-composition, unsupported, positive-value, and per-distance masks.",
  "controls": [
    "MC remains excluded",
    "artifact round-trip must exactly reproduce predictions",
    "composition-required counts must be nonzero",
    "DP/BFS never enters learner targets",
    "legacy aggregate MSE is secondary"
  ],
  "evaluation": "Certified-composition MAE/MSE plus positive-value, unsupported, per-distance, ranking, and calibration metrics; report counts and mean truth for each mask.",
  "experiments": [
    "smoke, leakage, mask, and artifact round-trip tests",
    "MLP TD1/TD4 at distances 16/32/64/128",
    "geometry and tabular TD1/TD4 at distance 128",
    "endpoint seed confirmation if runtime permits"
  ],
  "deliverables": [
    "full reusable artifacts",
    "corrected metric tables",
    "explicit list of unfinished cells"
  ],
  "compute": {
    "gpu_type": "L40S",
    "num_gpus": 1,
    "estimated_hours": 1
  }
}

{
  "proposal_id": "P-i981"
}

{
  "run_id": "R-j092",
  "section": "summary",
  "detail": "verification status, artifact round-trip, mask counts, and measured runtime"
}

{
  "run_id": "R-j092",
  "instruction": "Runtime cannot fit the full matrix. Preserve the MLP learner_td1 versus learner_td4 comparison at chain17/33/65/129 and all artifact exports. Shorten only the chain129 retry if needed. Defer geometry/tabular endpoint and extra seeds, and report them explicitly as unfinished; do not replace them with legacy metrics."
}

{
  "run_id": "R-j092",
  "section": "metrics",
  "detail": "corrected-mask MLP results, artifacts, and unfinished cells"
}

{
  "title": "Corrected composition masks reveal a single-seed TD4 advantage but leave architecture interaction open",
  "statement": "With evidence-defined composition-required masks and leakage-free learner targets, TD4 has lower composition MAE than TD1 at every tested chain endpoint. This does not settle overall long-horizon accuracy or architecture versus objective because the run is single-seed, the longest condition used a shorter retry, and geometry/tabular endpoint cells are missing.",
  "evidence": "Composition MAE TD1/TD4: chain17 0.1704/0.0783, chain33 0.1519/0.0679, chain65 0.1241/0.0649, chain129 0.1767/0.0608. Full predictions and transitions were saved and reloaded exactly; DP remained evaluation-only.",
  "caveats": "Raw MAE is affected by shrinking discounted values; chain129 used 20 epochs/60 episodes versus 50/120 elsewhere; geometry/tabular and three-seed confirmation were not completed."
}