Stepwise preference

One step, scored, with the correction a researcher would make.

1/9

File metadata

_format:“option-a-stepwise-critique”

_note:“One element per scored step. Element shape is identical to rlhf-data-example.json (traj_id, step_id, category, task, success, gt, history, current_step, after_step_observation, judge, score, correction) so an existing loader works unchanged. All `current_step`…”

traj_id: d7124e49-40b6-4cae-aaec-62182c100f89
step_id: 4
category: research
task: augmentor-enriched-logging-policy-with-safe-handoffs
success: null
gt: null
score: 2

history

user

Offline reinforcement learning aims to learn effective control policies from static, pre-collected datasets, which is attractive in real systems where online trial-and-error is costly or unsafe. A central difficulty is that performance is often bounded by the quality of the data-generating (logging) policy, yet in many real-world continuous-control settings these logging policies are hand-crafted, deterministic, scripted routines that reliably reach goals but produce long, redundant, and suboptimal trajectories. This work focuses on active positioning problems—tasks requiring precise placement of an object by an end-effector under partial observability and distorted movement dynamics—as a testbed. It investigates whether and how the data-generating process itself, rather than only the offline learning algorithm, can be improved to yield better data for offline RL. Research question I'm investigating: Can an augmentor trained on limited logged data be used during data collection to enrich a logging policy while ensuring smooth hand-offs back to the deterministic logging policy?

assistant

reasonGround the question in prior work on offline-RL data augmentation and behaviour-policy improvement before designing anything.

actionsearch_arxiv(query="offline reinforcement learning data augmentation behavior policy improvement trajectory stitching handoff continuous control", categories=["cs.LG","cs.RO"], date_from="2018-01-01", max_results=8, sort="relevance")

observation

8 results. Model-based Trajectory Stitching (Hepburn & Montana, 2022) connects a logged prefix to a higher-value suffix under a learned reachability/value check. ORIL (Zolna et al., 2020) covers learning from mixed-quality unlabeled experience. Lab findings #70 and #75 cover offline coverage/representation failure modes. Nothing found on improving the logging policy itself during collection.

current_step

assistant

reasonFormulate the augmentor as a bounded residual policy blended with the logger through an uncertainty gate, with an explicit handoff loss, and lay out a four-arm comparison.

actionrespond(summary="A clean formulation is a gated mixture a_t = (1-g_t) a_log(o_t) + g_t a_aug(h_t), with g_t in [0,1] a conservative gate based on augmentor uncertainty, deviation from logger-supported states, and estimated recoverability. Train the augmentor to produce bounded residual actions a_aug = a_log + clip(delta_theta, -eps, eps) rather than unconstrained control, then enforce a handoff loss over an H-step fade-out window: action mismatch, action jerk, and probability that the logger completes the task after takeover. Strongest matrix: (1) deterministic logger only, (2) action-noise logger, (3) residual augmentor without handoff objective, (4) uncertainty-gated residual augmentor with smooth handoff, all at matched interaction budget.")

after_step_observation

observation

Researcher: "Yeah I definitely think that better data can beat better algorithms: we should improve the logging policy. I think we should explore trajectory-level data augmentation"

judge

The question names the bottleneck explicitly: performance is bounded by the quality of the logging policy, and those loggers produce long, redundant, suboptimal trajectories. The agent answered a data problem with a controller. 1. Wrong level of intervention. A learned residual augmentor with an uncertainty gate introduces three new estimated objects (the residual policy, its uncertainty, the recoverability estimate) and each is a fresh failure mode, none of which is the thing that was wrong with the data. 2. The proposal buries the researcher's own framing. 'Better data can beat better algorithms' means edit the data-generating process; the gated mixture is a better algorithm wearing a collection costume. 3. It also throws away the property that makes the chosen setting worth studying. Deterministic dynamics and reward-as-a-function-of-state let you compute the value of an edit exactly; a learned uncertainty gate needs none of that and gains none of it. {"Score":2}

correction

error_type:“intervention_at_wrong_level”

first_problematic_step_id:4

critique:“Yeah I definitely think that better data can beat better algorithms: we should improve the logging policy. I think we should explore trajectory-level data augmentation”

preferred_insertion_point:4

research_principle:“When the stated bottleneck is data quality, intervene on the data-generating process at the granularity of a trajectory — do not answer it by bolting a learned controller with its own uncertainty estimator and auxiliary losses onto the logger.”