Stepwise preference
rlhf-stepwise.jsonOne step, scored, with the correction a researcher would make.
1/7
File metadata
_format:“option-a-stepwise-critique”
_note:“One scored step per element, matching the key structure used by rlhf-data-example.json. Current and preferred actions are condensed from the transcript so they remain reviewable; researcher critique text is verbatim except that no credential-bearing message is…”
- traj_id
- crl-importance-sampling-transcript-1
- step_id
- 1
- category
- research_design
- task
- collective-density-ratio-per-agent-ppo-value
- success
- null
- gt
- null
- score
- 2
history
user
train a crl critic on the PPO on-policy rollout ... dont do individual agent states, just do a critic predicting future collective state from current collective state. then, use this to do importance sampling on the whole buffer to estimate value function.
current_step
assistant
Proposed a collective reward and shared CRL value/advantage because the critic operates on collective state.
after_step_observation
observation
The researcher asked whether the density ratio could remain collective while the reward and resulting value stayed per-agent.
judge
The proposal collapsed two separate design choices. The transition-density model should be collective because the dynamics couple agents, but the PPO reward path is per-agent. The researcher specified the correct factorization: one collective W(S_i,S_j), then a per-agent matrix multiply against rewards. That preserves the intended CRL object without discarding credit assignment. The later implementation retained per-agent reward/advantage and passed discrete and continuous smoke tests. Score: 2.
correction
error_type:“unnecessary_credit_assignment_collapse”
first_problematic_step_id:1
critique:“Can the V_crl also be per-agent? meaning we can use crl critic to estimate the collective state density ratio, but then for a given state for the reward use the single agent reward? so the density ratio is still over the collective state but the reward is for…”
preferred_insertion_point:1
research_principle:“Separate the shared dynamics model from the per-agent learning signal.”