Pairwise preference

Two candidate continuations from one state, with an ordering.

1/11

File metadata

_format:“option-b-pairwise-preference”

_note:“DPO / reward-model shaped. Each pair is a REAL preference event unless flagged otherwise: `rejected` is what the agent actually emitted, `chosen` is what it actually emitted after the researcher's steer, and `critique` is the researcher's verbatim words (typos…”

pair_id: p01-improve-the-data-not-the-controller
synthetic_rejected: false
prompt: Give the first concrete formulation of the augmentor.
critique: Yeah I definitely think that better data can beat better algorithms: we should improve the logging policy. I think we should explore trajectory-level data augmentation
margin: strong

context

Opening turn. The researcher's question is whether an augmentor trained on limited logged data can enrich a hand-scripted, deterministic logging policy during collection while handing control back safely. Literature check surfaced Trajectory Stitching (Hepburn & Montana 2022) and ORIL. Nothing has been designed yet.

rejected

action:“respond”

summary:“A gated mixture at the action level: a_t = (1-g_t) a_log(o_t) + g_t a_aug(h_t), with g_t in [0,1] a conservative gate driven by augmentor uncertainty, deviation from logger-supported states, and estimated recoverability. Train the augmentor to emit bounded res…”

chosen

action:“ask_researcher”

summary:“Reframe the intervention as trajectory-level augmentation of the logged data and put the choice of augmentation to the researcher: hybrid rollouts (bounded augmentor segments inside logger rollouts with a smooth return to the logger), trajectory stitching (con…”

principle

When the stated bottleneck is data quality, intervene on the data-generating process at the granularity of a trajectory — do not answer it by bolting a learned controller with its own uncertainty estimator, gate, and auxiliary losses onto the logger.

downstream_evidence

The entire remainder of the session is trajectory-level: detour replacement, the summed-action bridge, the shortcut inequality. No learned augmentor network was ever trained, and the final method has no learned component at all — the researcher's later steer 'no network involved' is the same preference again.