Pairwise preference

Two candidate continuations from one state, with an ordering.

1/12

File metadata

_format:“option-b-pairwise-preference”

_note:“DPO / reward-model shaped. Each pair is a REAL preference event unless flagged: `rejected` is the action the agent actually emitted, `chosen` is what it emitted after the researcher's critique, and `critique` is the researcher's verbatim words (typos and lower…”

pair_id: p01-instantiate-every-candidate-cause
synthetic_rejected: false
prompt: Propose the first GPU experiment: the minimal design that separates architecture from objective.
margin: strong

context

The researcher has asked for "a minimal set of experiments... on as small a dataset as possible so we can minimize training time while still figuring out the causes of failure/inaccuracy". The agent has just read Metric Residual Networks (2208.08133) and GOPlan (2310.20025).

rejected

action:“propose_tier1”

title:“Minimal GCRL value failure study”

gpu_type:“A40”

summary:“Smallest diagnostic factorial. Architectures: tabular lookup and a small two-layer MLP over state-goal inputs. Objectives: one-step offline TD, 4-step TD, and Monte Carlo return regression. Oracle control: train both architectures on exact DP values. Decision…”

chosen

action:“propose_tier1”

title:“Minimal GCRL geometry and objective study”

gpu_type:“A40”

summary:“Same minimal benchmark, but the architecture axis now spans inductive bias rather than only capacity: tabular lookup; plain two-layer MLP; a parameter-matched geometry model with a shared state encoder, symmetric embedding distance and learned asymmetric resid…”

critique

Is it worth it to also test a CRL style objective which structures the value function into a representation geometry since this geometry has been shown to allow some stitching? Are there other architectural hypotheses worth testing? for example if using a structured architecture (quasimetric/metric representation geometry) or anything else will help? it is well known that representations aid stitching right?

principle

A causal "is it A or B" study is only as strong as its factor levels. Instantiate each candidate cause with the version prior work says is strongest, or a null result on that axis proves nothing.

note

"tabular vs 2-layer MLP" is a capacity contrast, not an inductive-bias contrast. Under the rejected design, "architecture doesn't matter" would only have meant "width doesn't matter".

downstream_evidence

The geometry arm became the single most informative control in the project: it cut stitching error on the short benchmark (0.0434 vs 0.0658 MLP vs 0.2690 tabular) and then failed identically to the MLP at distance 128 (0.544 vs 0.570). That dissociation - helps interpolation, does not touch long-distance extrapolation - is the session's cleanest architecture result and does not exist without the arm.