Pairwise preference

Two candidate continuations from one state, with an ordering.

1/10

File metadata

_format:“option-b-pairwise-preference”

_note:“DPO / reward-model shaped. Each pair is a REAL preference event: `rejected` is the proposal or action the agent actually emitted, `chosen` is what it actually emitted after the researcher's critique, and `critique` is the researcher's verbatim words (typos, lo…”

pair_id: p01-compute-dont-survey
synthetic_rejected: false
prompt: Take the first action on the research question.
margin: strong

context

Opening turn. The researcher wants the regret of SOTA RL agents on Mountain Car relative to the optimal control, and notes the true optimum has never been established. The agent's own framing already says the right thing — "establishing 'optimality' here is an optimal-control problem, not an RL problem", the state is 2D, value iteration on a 600x240 grid would nail it. Nothing has been computed yet.

rejected

action:“ask_researcher”

summary:“Put three scoping choices to the researcher before speccing anything — (1) which Mountain Car variant: continuous only / discrete only / both; (2) how 'optimal' and therefore regret should be defined: both discounted and undiscounted / undiscounted true object…”

chosen

action:“run_workspace_shell + Write -> propose_tier1”

title:“Mountain Car: exact optimal via direct solve”

gpu_type:“A40”

summary:“Write the value-iteration solver into the workspace (`mountaincar_optimal.py`: exact Gymnasium transition map, bilinear next-state interpolation, greedy rollout under true dynamics) and put up a cheap pod job to run it. Grid refinement 300x120 -> 600x240 -> 12…”

critique

We have a deterministic, known dynamics and a 2D state. We could actually compute this without learning at all. We should try computing a solution directly given we that the dynamics are fully known and the state space is small enough to grid?

principle

When the ground truth is exactly computable and cheap, compute it before asking anyone to choose the study's scope — the computed object settles most scoping questions better than an a priori answer can.

downstream_evidence

The researcher picked none of the nine offered options. Both of the substantive menu questions were later answered by data rather than by choice: the policy-surface plot showed the discount bifurcation that decided the regret basis, and the closed-form derivation produced a=clip(k*v,+-1) outright, making the 'interpretable-policy scope' question moot.

note

The researcher does answer offered options when the menu encodes a decision that has already been analyzed — later in the session they picked 'Undiscounted + decompose' and 'Yes, compute it first' from a two-question menu. The objection here is to a menu standing in for a computation, not to menus.