How close to optimality are current state-of-the-…

{
  "query": "MountainCarContinuous-v0 exact dynamics reward horizon optimal control value iteration regret",
  "categories": [
    "reinforcement-learning",
    "optimal-control"
  ],
  "max_results": 10
}

{
  "title": "Mountain Car exact optimum by value iteration, cross-checked",
  "research_question": "What is the trustworthy optimal control and return for MountainCarContinuous-v0 before any RL comparison is attempted?",
  "method": "Use global value iteration on the (x,v) grid with the exact deterministic transition and bilinear next-state interpolation as the primary solver. Refine from 300x120x21 through 600x240x41 to 1200x480x81. Cross-check with policy iteration on the grid and grid-free direct collocation from the canonical start. Compute the discounted agent objective and a near-undiscounted reference.",
  "controls": [
    "step-for-step transition equivalence to the benchmark",
    "Bellman residual convergence",
    "terminal +100 credit and absorbing-goal handling",
    "greedy rollout under exact dynamics versus predicted value",
    "policy-iteration agreement",
    "grid-free collocation agreement"
  ],
  "evaluation": "Agreement table across solvers and grid resolutions, with optimal return, rollout, effort, trajectory, value surface and policy surface.",
  "experiments": [
    "solver correctness tests before reporting values",
    "value iteration grid refinement",
    "policy iteration cross-check",
    "direct collocation cross-check",
    "persist a self-describing reference artifact"
  ],
  "deliverables": [
    "optimal return and numerical uncertainty",
    "value and policy arrays",
    "canonical trajectory",
    "reusable manifest"
  ]
}

{
  "proposal_id": "P-a13f"
}

{
  "run_id": "R-b24e",
  "section": "summary"
}

{
  "title": "Discounted Mountain Car optimum is numerically cross-validated",
  "statement": "For MountainCarContinuous-v0 from the canonical start, the gamma=0.99 optimum is approximately 47.81 discounted return and its exact-dynamics rollout scores 94.27 undiscounted.",
  "evidence": "Value iteration converges 47.57 to 47.67 to 47.713 across grid refinements; policy iteration differs by 7e-9 with 0% policy disagreement; grid-free collocation returns 47.807 in 66 steps. Dynamics, Bellman and terminal-credit controls pass.",
  "caveats": "The gamma=0.9999 result, 98.18 undiscounted in 165 steps, is only a lower bound on the 999-step undiscounted optimum."
}

{
  "title": "Plot the optimal policy surface under discounted and min-effort regimes",
  "research_question": "Does changing the return objective merely retune one controller, or select a qualitatively different policy geometry?",
  "method": "Recompute the two policy surfaces on a 600x240 state grid with 41 actions. Render action heatmaps with zero-action switching curves and overlay the exact canonical-start trajectory.",
  "controls": [
    "same dynamics and state grid for both objectives",
    "mark the goal region",
    "report saturated versus graded state-space fraction"
  ],
  "evaluation": "Direct visual and numerical comparison of switching structure, saturation and canonical trajectory.",
  "experiments": [
    "gamma=0.99 policy surface",
    "gamma=0.9999 policy surface",
    "canonical trajectory overlay"
  ],
  "deliverables": [
    "side-by-side policy surfaces",
    "switching-curve characterization",
    "trajectory overlay"
  ]
}

{
  "proposal_id": "P-c35a"
}

{
  "run_id": "R-d46b",
  "section": "qualitative",
  "detail": "policy geometry, saturation and trajectory"
}

{
  "title": "Discounting changes Mountain Car policy topology",
  "statement": "The discounted optimum is a mostly saturated boomerang switching surface, whereas the near-undiscounted policy is a smooth velocity-organized force field.",
  "evidence": "On the common 600x240x41 solve, gamma=0.99 is 59% saturated and reaches in 66 steps; gamma=0.9999 is 99% graded. Switching-curve RMS |v| is 0.021 versus 0.008.",
  "caveats": "The near-undiscounted surface is not itself the exact 999-step finite-horizon optimum."
}

{
  "query": "Mountain Car validated dynamics discounted optimum policy geometry and undiscounted lower bound",
  "max_results": 5
}

{
  "title": "Validate the closed-form minimum-effort law a = clip(k v)",
  "research_question": "Is the finite-horizon undiscounted optimum essentially the one-parameter velocity-feedback controller predicted by the energy argument, and what residual does time variation explain?",
  "method": "Sweep constant gains under exact dynamics to find the feasibility cliff within the 999-step cap. Independently solve finite-horizon backward dynamic programming at gamma=1 and execute the time-indexed greedy policy under exact dynamics. Compare actions along the optimal trajectory with velocity and position.",
  "controls": [
    "step-for-step exact dynamics",
    "grid refinement",
    "execute the time-varying policy rather than a stationary interpolation of V_999",
    "separate injected-work energy identity from endpoint-energy discretization error",
    "test terminal coast, action cap and two-phase gain only if needed"
  ],
  "evaluation": "Undiscounted return, effort, steps, gain threshold, gap to dynamic programming, action-velocity and action-position correlations, analytic effort/growth/period checks.",
  "experiments": [
    "constant-gain feasibility sweep",
    "finite-horizon backward-DP confirmation",
    "exact-dynamics time-varying rollout",
    "analytic scaling tests",
    "minimal augmentation checks"
  ],
  "deliverables": [
    "true undiscounted target",
    "closed-form controller and gap",
    "objective-mismatch baseline",
    "durable reference arrays"
  ]
}

{
  "proposal_id": "P-e57c"
}

{
  "run_id": "R-f68d",
  "section": "metrics"
}

{
  "run_id": "R-f68d",
  "section": "controls"
}

{
  "title": "The undiscounted optimum is near-analytic velocity feedback",
  "statement": "The true 999-step undiscounted optimum is approximately 99.62. A constant-gain law a=clip(kv) reaches 99.452 and captures 99.8% of it; the remaining 0.167 comes from the optimal time-varying gain.",
  "evidence": "Refined DP values 99.605 and 99.617; exact-dynamics time-varying rollout 99.6195 with effort 3.805 in 968 steps. Constant gain returns 99.452 with effort 5.48 in 933 steps. Optimal-trajectory corr(a,v)=0.86 and corr(a,x)=0.06.",
  "caveats": "The gain threshold is razor-thin and reported k=4.9945 is rounded; the continuous-time law is near-optimal rather than exactly equal to the discrete finite-horizon controller. A stationary greedy policy is not a valid executable substitute for the time-indexed finite-horizon policy."
}

{
  "query": "Mountain Car undiscounted optimum constant-gain law mismatch baseline and policy fingerprint",
  "max_results": 5
}

{
  "title": "Do solving RL agents implement velocity feedback?",
  "research_question": "Before designing causal sweeps, do trained Mountain Car solvers have the same a approximately k*v geometry as the near-analytic optimum, or a fundamentally different policy field?",
  "method": "Train SAC, TD3, PPO and TQC at gamma=0.99 with 10 seeds each. Evaluate deterministic undiscounted return. For each solver, sample the action over the full state grid and render it beside a=clip(k*v), with a residual map and canonical trajectory overlay. Fit effective gain and compute full-grid action-velocity and action-position correlations, velocity-fit R2 and saturation.",
  "controls": [
    "analyze non-reaching seeds separately from solver regret",
    "use full-grid correlations to disentangle state collinearity along trajectories",
    "same undiscounted evaluation and reference targets for every algorithm",
    "inspect policy portraits rather than selecting on return alone"
  ],
  "evaluation": "Solver counts and undiscounted returns versus 99.62 and 94.27, plus full-grid policy fingerprints that decide velocity feedback versus different topology.",
  "experiments": [
    "train and evaluate four algorithms across 10 seeds each",
    "regret plot with failures marked separately",
    "policy portraits for every solver",
    "full-grid fingerprints and structural verdict"
  ],
  "deliverables": [
    "per-seed regret table",
    "policy portraits and residual maps",
    "fingerprint summary",
    "plain-language verdict that scopes the next experiment"
  ]
}

{
  "proposal_id": "P-g79e"
}

{
  "run_id": "R-h80f",
  "section": "qualitative",
  "detail": "first completed solver: compare trajectory correlations with the full policy surface"
}

{
  "run_id": "R-h80f",
  "instruction": "For every solving policy, compute corr(a,x), corr(a,v), the velocity-fit R2 and effective gain over the full rectangular state grid, not only along the rollout. Keep trajectory-based correlations as secondary visitation diagnostics. Regenerate portraits and the summary using surface-based fingerprints, because the preliminary SAC trajectory has x-v collinearity that inflates corr(a,x). Do not change training or the evaluation return."
}

{
  "run_id": "R-h80f",
  "section": "metrics"
}

{
  "run_id": "R-h80f",
  "section": "qualitative",
  "detail": "full-grid policy fingerprints, failure modes and representative portraits"
}

{
  "title": "Solving Mountain Car agents share the optimal velocity-feedback topology",
  "statement": "Across SAC, TQC, TD3 and PPO solvers, the learned full-grid policy is velocity-organized rather than position-dependent. The reported-return gap is primarily gain and saturation, not representational inability.",
  "evidence": "40 runs produced 33 solvers: SAC 10, TQC 10, TD3 9, PPO 4. Surface corr(a,x) is 0.03, 0.08, -0.07 and -0.29 respectively, while corr(a,v) is about 0.8. Most solver returns are 93.5-94.5; lower-gain PPO seeds reach 96-97.",
  "caveats": "One environment, one training discount and tuned library configurations. Non-reaching seeds are a separate failure mode. Trajectory correlations are confounded by x-v collinearity and must not be used to infer policy topology."
}

{
  "run_id": "R-h80f",
  "reason": "All 40 planned training runs, policy portraits, surface fingerprints and persisted artifacts are complete and verified; release compute."
}