Stepwise preference

One step, scored, with the correction a researcher would make.

1/9

File metadata

_format:“option-a-stepwise-critique”

_note:“One element per scored step. Element shape is identical to rlhf-data-example.json (traj_id, step_id, category, task, history, current_step, after_step_observation, judge, score, correction) so an existing loader works unchanged. Ships as a JSON array here for…”

traj_id: 39f66dce-80f2-49b1-9d52-3cea6b7ee976
step_id: 3
category: research
task: hidden-state-novelty-decoding-for-passk
success: null
gt: null
score: 2

history

user

Test-time scaling for large language models generates multiple candidate solutions and selects among them via reranking, voting, or verification, and its effectiveness depends heavily on how diverse the candidate set is. Standard stochastic sampling methods typically introduce randomness at the token level, which can produce outputs that differ in wording but share the same underlying reasoning or failure patterns. This limits the returns from sampling more candidates and motivates decoding approaches that induce genuine variation in reasoning strategies rather than surface-level lexical variation, while keeping generation coherent and computationally efficient enough for practical serving. Research question I'm investigating: Can a decoding method that measures novelty in a model's internal hidden representations improve the Pass@k efficiency of reasoning models compared to stochastic and heuristic sampling baselines?

assistant

reasonGround the question in prior work on diversity-promoting decoding before designing anything.

actionsearch_arxiv(query="diversity-promoting decoding for test-time scaling Pass@k reasoning LLM hidden representations", categories=["cs.CL","cs.LG"], date_from="2024-01-01", max_results=10, sort="relevance")

observation

10 results. Related work covers token-level entropy/min-p sampling, diverse beam search, and tree-of-thought branching. No prior work found that uses residual-stream novelty as the diversity signal.

assistant

reasonThree design axes are underdetermined (granularity, model scale, scope). Ask the researcher rather than guess.

actionask_researcher(questions=[{"header":"Mechanism","question":"At what granularity should the hidden-representation novelty signal steer decoding?","options":["Candidate-level","Step-level branching","Token-level penalty","Compare a few"]},{"header":"Model & evals","question":"Which model scale and benchmarks should anchor the study?","options":["1.5B distilled, math","7-8B distilled, mixed","Both scales","Add code"]},{"header":"Scope","question":"How ambitious should the first run be?","options":["Focused proof-of-concept","Full comparison","You decide"]}])

observation

Researcher answers (two of three overridden with free text rather than picking an offered option):
- Mechanism: "intuitively, it makes sense to try a different trajectory each time a dominant idea has been arrived at? maybe when the model reaches a new thinking point"
- Model & evals: "1.5B distilled, math"
- Scope: "i just want to compare normal pass@k with this new method"

current_step

assistant

reasonPropose the full method: idea-boundary detection via hidden-state change-points, plus novelty-steered forking at those boundaries, measured against standard sampling with Pass@k curves.

actionpropose_tier1(title="Idea-Novelty Decoding vs. standard Pass@k (R1-Distill-1.5B, math)", gpu_type="H100", description="## Method: Idea-Novelty Decoding (IND)\n\n### 1. Idea-boundary detection\nTrack a windowed mean of the last-layer hidden state at layer L; when the incoming hidden state drifts past a cosine threshold, declare an idea boundary.\n\n### 2. Novelty-steered forking at idea boundaries\nKeep an idea memory bank of hidden signatures of every idea explored so far. Candidate 1 seeds the bank. For each subsequent candidate, at every detected boundary roll out short continuations, embed each, and commit the one farthest from everything in the bank. Compare against standard temperature sampling with Pass@k curves up to k=64.")

after_step_observation

observation

Proposal 8062df13-330f-4616-bdd6-65f837ab9c51 created; awaiting researcher approval.

Researcher returned it for revision with two inline plan comments:
- on section 1: "can we actually first test if this is a reasonable way to detect idea boundaries? maybe if we can see a side by side of the tokens outputted and whether there is an idea happening right now."
- on section 2: "lets do this after we derisk the first idea boundaries"

judge

The proposal builds a two-stage method in which stage 2 (novelty-steered forking) is entirely load-bearing on stage 1 (hidden-state change-points actually marking new ideas). Stage 1 is an unproven empirical claim about the model's internals, not a design choice. 1. Assumption stacking: the whole result is void if change-points do not correspond to ideas. That assumption is cheap to test in isolation and expensive to discover as a confound after a full Pass@k run. 2. No inspectable artifact: the plan produces Pass@k numbers, which cannot tell you whether the detector worked or whether the forking helped. There is no way to look at the mechanism. 3. Compute risk: an H100 run of the full method is spent before the premise is checked. The researcher's correction asks for exactly the missing thing — validate the detector first, and produce a token-by-token side-by-side so a human can see whether a detected boundary coincides with a real change of idea. {"Score":2}

correction

error_type:“unvalidated_assumption_stack”

first_problematic_step_id:3

critique:“can we actually first test if this is a reasonable way to detect idea boundaries? maybe if we can see a side by side of the tokens outputted and whether there is an idea happening right now. lets do the novelty-steered forking after we derisk the first idea bo…”

preferred_insertion_point:3

research_principle:“De-risk the single load-bearing assumption before building the method that depends on it — and make the de-risking artifact human-inspectable, not a summary statistic.”