Stepwise preference

One step, scored, with the correction a researcher would make.

1/8

File metadata

_format:“option-a-stepwise-critique”

_note:“One element per scored step. Element shape is identical to rlhf-data-example.json (traj_id, step_id, category, task, history, current_step, after_step_observation, judge, score, correction) so an existing loader works unchanged. Ships as a JSON object with a s…”

traj_id: 15e7078e-15d5-4b4a-8923-2444ce09d0a9
step_id: 15
category: research
task: successor-measure-rl-for-novel-open-world-capabilities
success: null
gt: null
score: 2

history

user

I'm curious about how RL can help LLMs get better at doing fully novel stuff. Let's scope out a small experiments/couple of things so we can figure out some promising directions

assistant

reasonCheck the lab knowledge base for anything prior before scoping.

actionglob(pattern="knowledge/**/*.md")

observation

No matches. The knowledge base is empty — nothing prior to build on.

assistant

reasonFrame the central tension in the literature — RLVR reweights the base distribution (Yue et al.) vs prolonged-RL expands it — and offer three cheap diagnostics: (A) pass@k crossover before/after RLVR, (B) exploration pressure as the lever, (C) transfer to a held-out task family. Ask the researcher which to formalize and on what task.

actionask_researcher(questions=[{"header":"Direction","question":"Which direction should I formalize into a GPU experiment first?","options":["A: sharpen vs expand","B: exploration lever","C: transfer to held-out","A then B combined"]},{"header":"Task","question":"What task/reward setting do you prefer?","options":["Math (verifiable)","Code (unit tests)","Puzzle/game (novel)"]}])

observation

Researcher picked from the offered options: Direction = "B: exploration lever"; Task = "Puzzle/game (novel)".

current_step

assistant

reasonCountdown is verifiable and a 1.5B base model is weak at it, so there is real novelty headroom. Formalize the exploration-mechanism sweep: four arms, two seeds, pass@k to 256, crossover = sharpen vs high-k dominance = expand.

actionpropose_tier1(title="Exploration pressure: does RL expand or just sharpen? (Countdown)", gpu_type="A100", description="GOAL. Test whether explicit exploration pressure lets RL EXPAND a small LLM's capability boundary on a novel puzzle task, versus merely SHARPENing the base distribution. Testbed: Countdown (reach a target integer from N given numbers using +,-,*,/; verifiable exact-match reward, no reward model needed).\n\nDIAGNOSTIC. The discriminator is pass@k for k=1..256 on a held-out set, frozen base vs each trained arm. Crossover => sharpening. High-k dominance => genuine expansion.\n\nMODEL / ALGO. Qwen2.5-1.5B base, GRPO with binary verifiable reward, compute-matched across arms.\n\nINDEPENDENT VARIABLE = exploration mechanism (4 arms): 1. Vanilla GRPO (the sharpen control). 2. + Entropy bonus. 3. + High rollout temperature (1.2) during training only. 4. + Diversity/novelty reward for correct solutions whose canonicalized expression differs from others in the GRPO group. 2 seeds per arm => 8 runs.\n\nMETRICS. Primary: pass@k curves. Secondary: distinct valid solutions per problem, policy entropy over training.")

after_step_observation

observation

Proposal created; awaiting researcher approval.

The researcher did not approve it. His next message: "hmm is there something more RL-centric that can solve something more concrete there instead of training an LLM?"

[The proposal was never revisited; the agent's own later accounting lists it as dead — "the LLM Countdown 'sharpen vs expand' study (you pivoted away from LLMs)".]

judge

The design is internally clean — compute-matched arms, one independent variable, a genuine discriminator in the pass@k crossover — and it is still the wrong experiment, because the substrate cannot support the claim. 1. Confounded substrate. The question is whether RL makes an agent discover behaviour it did not already have. In a pretrained LLM, any high-k gain is entangled with pretraining priors over arithmetic, with formatting/parsing of the expression, and with reward shaping. A positive result would not be attributable to the exploration mechanism, and a null would not be either. 2. The agent had the counter-argument in hand. One turn later, unprompted, it wrote that "the LLM version tangles the novelty question up with pretraining priors, tokenizer quirks, and reward-shaping hacks, so it's hard to cleanly attribute any 'expansion' to RL itself" — and called the RL-native version "the sharper move". It could have applied that reasoning before spending the proposal. 3. Cost asymmetry. Eight GRPO runs sampling to k=256 is an expensive way to ask a question that an RL-native environment answers in a few GPU-hours with an unambiguous, countable definition of novel behaviour (which deep achievements does an arm unlock that the baseline never reaches). The researcher's redirect is not a change of topic; it is a demand that the phenomenon be isolable. {"Score":2}

correction

error_type:“confounded_substrate”

first_problematic_step_id:15

critique:“hmm is there something more RL-centric that can solve something more concrete there instead of training an LLM?”

preferred_insertion_point:15

research_principle:“Run the mechanism question in the smallest substrate that still exhibits the phenomenon; if the substrate brings its own explanations for the effect you are measuring, no result from it is attributable.”