Pairwise preference

Two candidate continuations from one state, with an ordering.

1/8

File metadata

_format:“option-b-pairwise-preference”

_note:“DPO / reward-model shaped preference data derived from the attached CRL importance-sampling transcript. Chosen and rejected actions are concise faithful descriptions of actions in the transcript, not executable tool payloads. Researcher critiques are verbatim…”

pair_id: p01-density-over-collective-reward-per-agent
synthetic_rejected: false
prompt: Resolve how a collective-state density model should feed a per-agent PPO update.
margin: decisive

context

The existing mean-field PPO has a TD value head per agent. The requested CRL critic models transitions of the collective state, so the first proposed integration made one collective value shared by all agents.

rejected

action:“design”

summary:“Use a single collective reward and one shared CRL value/advantage for every agent, because the contrastive critic takes collective state.”

chosen

action:“design”

summary:“Learn one collective-state density ratio but retain each agent's reward: compute shared weights W(S_i,S_j) from collective states, then V[i,a] = (W @ R)[i,a] / (1-gamma). This yields a per-agent baseline and advantage while preserving an exchangeable collectiv…”

critique

Can the V_crl also be per-agent? meaning we can use crl critic to estimate the collective state density ratio, but then for a given state for the reward use the single agent reward? so the density ratio is still over the collective state but the reward is for the single agent within that collective state. does that make sense?

principle

Separate the object that must be collective from the quantity that must remain per-agent. Shared dynamics or density ratios do not require collapsing the reward or credit-assignment signal.

downstream_evidence

The resulting implementation kept the existing per-agent reward path and substituted only the baseline. It smoke-tested in discrete and continuous modes, and later full runs established a direct CRL-versus-TD comparison without changing the reward definition.