Synthetic SFT

Multi-turn tool-using investigation as a clean demonstration.

id: ml_research_trajectory_open_world_zero_shot_rl_001
domain: reinforcement_learning_research
task_type: multi_turn_autonomous_research_investigation
variant: A_reasoning_externalized

provenance

source:“human-steered research session, 2026-07-14”

construction:“The human researcher's steering turns and the orchestrator's execution turns were collapsed into a single assistant agent. The source contained nine proposal records, but several were revisions or infrastructure-driven duplicates; they are represented as one f…”

note:“This is a deliberately thin, proposal-led trajectory. The only launched study was stopped before its main comparison finished. Assistant turns state design criteria before the proposal and never turn unlaunched proposals, smoke tests, or a single completed cel…”

tools

Functions available to the assistant

8 tools

01search_literature
Search the research literature and return relevant papers with findings.
querystring*categoriesstring[]date_fromstringmax_resultsinteger
02search_findings
Search findings previously recorded by this lab, to reuse established results instead of re-deriving them.
querystring*max_resultsinteger
03propose_experiment
Draft a runnable experiment specification. Returns a proposal id and a compute estimate. Does not run anything.
titlestring*research_questionstring*methodstring*controlsstring[]evaluationstring*experimentsstring[]*deliverablesstring[]computeobject
04launch_experiment
Launch a proposed experiment on compute. Returns a run id. The run executes autonomously.
proposal_idstring*
05read_experiment_results
Read output from a running or finished experiment.
run_idstring*sectionstring*detailstring
06steer_experiment
Send an instruction to a running experiment to change what it does mid-flight.
run_idstring*instructionstring*
07stop_experiment
Cancel a running experiment and release its compute.
run_idstring*reasonstring*
08record_finding
Record a durable finding so later experiments can build on it without re-deriving it.
titlestring*statementstring*evidencestring*caveatsstring

* required

messages

system

You are an autonomous ML research agent with access to compute. You design, run, and interpret experiments to answer research questions. Work the way a good empirical researcher works. Before building anything, identify the assumption the whole idea rests on and test that first. Prefer the experiment that could kill the idea over the experiment that would showcase it. Do not pre-commit to experiment N+1 before seeing the result of experiment N. When a measurement could be produced by an artifact, run the control that would expose the artifact, and select on the controlled quantity rather than on how good the raw result looks. State negative results plainly and diagnose them mechanistically. Reuse prior results instead of re-spending compute on settled questions, and hold the evaluation set fixed across conditions so comparisons remain valid. Do not invent numbers: every quantitative claim must come from a tool result.

user

I'm curious about how RL can help LLMs get better at doing fully novel stuff. Let's scope out a small experiment or a couple of things so we can figure out some promising directions.

assistant

The phrase "fully novel" is too broad to spend compute against. It could mean that RL raises the probability of a behavior already latent in a pretrained model, that exploration discovers states a reward-only agent never visits, or that a learned representation composes known skills into a new goal. Those are different scientific questions. Before choosing a substrate, I want to know whether this lab already has a settled result I should build on.

search_findings()

{
  "query": "RL novelty exploration successor measures goal-conditioned open-world control",
  "max_results": 10
}

toolsearch_findings

{"results":[],"notice":"No prior lab findings match this question."}

assistant

No prior result narrows it. I am not using an LLM as the first instrument. A language-model experiment would tangle the causal question with pretraining support, tokenization, and reward-formatting choices: an apparent novelty gain could still be redistribution inside the base distribution. A native RL environment lets novelty mean something operational—coverage of new states, attainment of previously unsupported goals, or composition across a goal hierarchy. Minecraft is also the wrong first substrate. It matches the ambition but adds perception, world-server, and large-scale data engineering before the representation question is identified. The useful cheap proxy is an open-ended tech-tree environment where goal depth, partial observability, and procedural variation survive but the systems burden does not. I need literature to answer three design questions before proposing anything: whether successor measures already support reward-free zero-shot goals, whether recent methods address low-coverage data, and whether partial observability invalidates the obvious feedforward design.

search_literature()

{
  "query": "forward-backward successor measures zero-shot goal-conditioned reinforcement learning low-quality data Craftax Minecraft",
  "categories": [
    "cs.LG",
    "cs.AI"
  ],
  "date_from": "2023-01-01",
  "max_results": 10
}

search_literature()

{
  "query": "zero-shot reinforcement learning partial observability recurrence HILP Craftax JAX implementation",
  "categories": [
    "cs.LG",
    "cs.AI"
  ],
  "date_from": "2024-01-01",
  "max_results": 10
}

toolsearch_literature

{"results":[{"title":"Does Zero-Shot Reinforcement Learning Exist? / controllable_agent","finding":"Forward-Backward representations learn from reward-free data and can optimize a reward specified at test time without additional learning.","code":"official PyTorch implementation available"},{"title":"Zero-Shot Reinforcement Learning from Low Quality Data","venue":"NeurIPS 2024","finding":"Measure-Conservative FB and Value-Conservative FB suppress unsupported actions and are designed for inexhaustive or low-quality pretraining data.","code":"released implementation targets state-based control benchmarks"},{"title":"Foundation Policies with Hilbert Representations","venue":"ICML 2024","finding":"HILP learns a distance-preserving temporal representation for long-horizon zero-shot control from unlabeled data.","code":"official zero-shot implementation is PyTorch"},{"title":"Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning","finding":"Hierarchical switching composes successor-measure policies without fixed horizons or manually designed subgoals."}],"notice":"No result found applying FB or HILP to Crafter, Craftax, or Minecraft."}

toolsearch_literature

{"results":[{"title":"Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning","venue":"ICML 2024","finding":"Craftax-Classic is a JAX rewrite of Crafter reported up to 250x faster; a one-billion-interaction PPO run finishes in under an hour on one GPU. The environment requires exploration, planning, and memory."},{"title":"Zero-Shot Reinforcement Learning Under Partial Observability","finding":"Standard Forward-Backward and successor-feature approaches degrade under partial observability; recurrent memory substantially improves them."},{"title":"JaxGCRL","finding":"Provides JAX goal-conditioned infrastructure but does not implement FB, successor features, or HILP."}],"implementation_check":"No existing JAX implementation of FB or HILP zero-shot RL was found; the HILP zero-shot requirements list torch and torchvision."}

assistant

The literature changes the admissible design. Craftax keeps the open-world structure while making a sweep plausible, but it is a POMDP, so a feedforward successor representation would confound method failure with missing state. Partial observability is not a nuisance variable here; memory must be in the main models and removed only at a controlled reference cell. MC-FB is the sensible FB variant because the question explicitly includes weak exploration data. HILP is useful as a second representation family, but not as the headline. I do not care which acronym wins; I care whether the conclusions about coverage, capacity, and memory survive a change of representation. That distinction cuts scope. The main study gets two actual research questions. First: as representation capacity and reward-free data coverage vary, which one binds the deepest zero-shot goal that remains reachable? Second: does the value of recurrence grow with goal depth? The tempting method-comparison question is demoted to a robustness check, because it does not answer how to scale goal-conditioned RL. Two attractive companion studies stay on paper. Hierarchical composition could test whether shallow goal directions chain into deeper behavior, and held-out procedural worlds could test whether the representation generalizes rather than memorizes. Both require a characterized recurrent MC-FB checkpoint under the stronger coverage source. Launching them now would make them depend on an unvalidated backbone and turn one uncertainty into three runs. I will not pre-commit that compute. Implementation risk needs its own gate. The available reference code is PyTorch while the launched environment stack is JAX. Any port must pass end-to-end smoke tests for FB, HILP, and the non-representation baseline before the sweep is interpreted. That does not prove the algorithms are faithful to their papers, but it prevents a broken training or evaluation path from masquerading as an open-world null. Finally, a zero on an achievement absent from pretraining is not a representation failure. It is a support failure. Therefore every result must report coverage by achievement, comparisons on the matched set of supported goals, and the random-versus-explorer contrast before any claim about representation dimension.

assistant

propose_experiment()

{
  "title": "Coverage, capacity, and memory limits of zero-shot goal-conditioned RL in Craftax",
  "research_question": "Does reward-free representation learning retain zero-shot goal reach as a partially observed tech tree gets deeper, and is the binding constraint representation capacity, exploration coverage, or memory? MC-FB and HILP are parallel representation families used to test robustness of those conclusions, not a headline winner-take-all comparison.",
  "method": "Use Craftax-Classic with symbolic partial observations. Learn MC-FB and HILP representations from reward-free trajectories. Main models use a recurrent encoder. Vary representation dimension over 64, 128, and 256 and data source between a random policy and an RND explorer. FB, HILP, and GCBC share identical data within each coverage arm. Before the main sweep, require end-to-end smoke tests for all three training and goal-evaluation paths. At test time evaluate zero-shot achievement goals without per-goal representation training.",
  "controls": [
    "GCBC on the identical reward-free data as a non-successor-representation control",
    "feedforward memory-off ablation only at the d=128 RND reference cell",
    "per-goal PPO-RNN oracle on a shallow, middle, and deep subset as a reachability upper bound",
    "achievement-level data-coverage table and comparisons restricted to the matched supported-goal set"
  ],
  "evaluation": "Report zero-shot success per achievement and by tech-tree depth, success as a function of representation dimension and data coverage, and the memory-on/off gap by depth. Do not interpret an unsupported deep-goal zero as evidence against a representation method. Held-out reward vectors are secondary.",
  "experiments": [
    "Preflight: inspect the environment API, measure random-policy achievement support, and smoke-test MC-FB, HILP, and GCBC end to end",
    "Main recurrent sweep: method by representation dimension by exploration coverage",
    "Memory control: recurrent versus feedforward at the fixed reference cell",
    "Matched-support analysis and shallow/middle/deep oracle checks"
  ],
  "deliverables": [
    "achievement coverage table by data source",
    "success-versus-depth curves",
    "capacity-by-coverage phase diagram",
    "memory-gap-by-depth analysis",
    "written verdict that separates support failure from representation failure"
  ],
  "compute": {
    "gpu_type": "A100-80GB",
    "num_gpus": 1
  }
}

toolpropose_experiment

{"proposal_id":"P-c7a1","status":"drafted","compute":{"gpu_type":"A100-80GB","num_gpus":1},"scope":"one main study; hierarchy and procedural-generalization companions deferred"}

assistant

launch_experiment()

{
  "proposal_id": "P-c7a1"
}

toollaunch_experiment

{"run_id":"R-f2b4","status":"running","title":"Coverage, capacity, and memory limits of zero-shot goal-conditioned RL in Craftax"}

assistant

The first read is not for a leaderboard. It is for the load-bearing support measurement and the smoke gates. If random play never reaches the deep tech tree, the random arm is a negative coverage control rather than a fair substrate for deciding between representations.

read_experiment_results()

{
  "run_id": "R-f2b4",
  "section": "controls",
  "detail": "environment ground truth, random-policy achievement coverage, collected-dataset support, evaluator checks, and pipeline smoke status"
}

toolread_experiment_results

{"environment":{"name":"Craftax-Classic-Symbolic","observation_shape":[1345],"actions":17,"achievements":22},"random_policy_probe":{"rollouts":512,"steps_per_rollout":256,"achievements_with_nonzero_coverage":9,"unlock_fraction":{"WAKE_UP":1.0,"COLLECT_WOOD":0.6992,"COLLECT_SAPLING":0.6016,"PLACE_PLANT":0.4785,"PLACE_TABLE":0.2129,"COLLECT_DRINK":0.2012,"MAKE_WOOD_PICKAXE":0.0254,"MAKE_WOOD_SWORD":0.0137,"EAT_COW":0.002,"DEFEAT_ZOMBIE":0.0,"COLLECT_STONE":0.0,"PLACE_STONE":0.0,"EAT_PLANT":0.0,"DEFEAT_SKELETON":0.0,"MAKE_STONE_PICKAXE":0.0,"MAKE_STONE_SWORD":0.0,"PLACE_FURNACE":0.0,"COLLECT_COAL":0.0,"COLLECT_IRON":0.0,"COLLECT_DIAMOND":0.0,"MAKE_IRON_PICKAXE":0.0,"MAKE_IRON_SWORD":0.0},"depth_note":"Every achievement at depth 3 or deeper was absent."},"reward_free_dataset":{"rollouts":1024,"steps_per_rollout":256,"supported_achievements":8,"support_depth":"all supported achievements were at depth 2 or shallower"},"evaluation_correction":"The evaluator originally accumulated achievements across automatic resets; it was corrected to count only the first episode so a later episode cannot be credited to the goal-conditioned rollout.","smoke_status":{"MC-FB":"passed end to end","HILP":"passed end to end","GCBC":"passed end to end"},"main_sweep_status":"started but incomplete"}

assistant

The first runtime measurement has already changed the interpretation. Random play touched only a small shallow subset, and the collected buffer is narrower still. Everything at depth three or beyond is outside the probe's support. Dimension cannot manufacture support: increasing the rank of a successor factorization cannot teach occupancy of states that never appear. A deep-goal zero in the random arm is therefore expected and scientifically uninformative about MC-FB versus HILP. This is not yet proof that coverage is globally the binding constraint. It is proof that coverage binds the random-data arm. The RND arm still has to demonstrate that it actually moves the frontier; until then, saying "exploration solves it" would be speculation. The evaluator correction is equally important. Crediting achievements across automatic resets would turn multiple episodes into one apparent rollout and inflate goal success. Because that bug was found before the main comparison and the corrected evaluator is what passed the smoke tests, it is implementation hygiene rather than a scientific failure. I am steering on the new evidence, not rewriting a bad proposal. Treat random data as a support-limited control; compare methods only on the matched set of supported goals; prioritize the RND coverage arm before spending the remaining budget filling out representation dimensions; retain the memory control at its fixed reference cell. Do not launch hierarchy or procedural-generalization companions while that reference cell is still uncharacterized.

steer_experiment()

{
  "run_id": "R-f2b4",
  "instruction": "Use the measured support frontier to change execution priority. Keep the random-policy condition as a coverage-failure control and report its deep-goal zeros as unsupported, not as representation failures. Compare MC-FB, HILP, and GCBC on the identical supported-goal set. Prioritize completing and measuring the RND coverage arm before filling the remaining representation-dimension cells. Keep the memory-off ablation confined to the fixed reference cell. Do not start the hierarchy or procedural-generalization companions."
}

toolsteer_experiment

{"run_id":"R-f2b4","accepted":true,"analysis_policy":{"random_arm":"support-limited control","comparison":"matched supported goals","priority":"RND coverage before additional capacity cells","memory_ablation":"reference cell only","companion_runs":"deferred"}}

assistant

Now I want the narrowest honest progress read: completed main-sweep cells and their exact caveats. Smoke-test numbers do not count as study results.

read_experiment_results()

{
  "run_id": "R-f2b4",
  "section": "metrics",
  "detail": "completed non-smoke cells only; identify every planned axis still missing"
}

toolread_experiment_results

{"random_policy_baseline":{"evaluation_horizon":200,"depth_success":{"0":0.45703125,"1":0.1640625,"2":0.001953125,"3":0.00390625,"4":0.00078125,"5":0.0,"6":0.0},"caveat":"This baseline aggregates over all goals, whereas learned methods were evaluated only where the buffer contained goal representatives; direct aggregate comparison is unmatched."},"completed_representation_cell":{"method":"MC-FB","representation_dim":64,"coverage":"random","recurrent":true,"seed":0,"training_steps":3000,"evaluation_envs":256,"evaluation_horizon":200,"depth_success":{"0":0.1845703125,"1":0.017578125,"2":0.0},"per_goal_success":{"COLLECT_WOOD":0.03515625,"PLACE_TABLE":0.0,"COLLECT_SAPLING":0.13671875,"COLLECT_DRINK":0.0,"MAKE_WOOD_PICKAXE":0.0,"MAKE_WOOD_SWORD":0.0,"PLACE_PLANT":0.03515625,"WAKE_UP":0.56640625},"train_time_seconds":52.28421950340271,"total_time_seconds":93.6250650882721},"missing":["all HILP main cells","all GCBC main cells","RND coverage cells","remaining representation dimensions","second seeds","memory ablation","oracle checks","hierarchical composition","procedural generalization"],"status":"partial; no scaling or method-comparison verdict is available"}

assistant

One completed cell is not a sweep. The MC-FB cell shows that the training and evaluation machinery can produce nonzero goal-conditioned behavior on supported shallow goals. It does not establish that MC-FB beats HILP, that larger representations help, that recurrence matters, or that RND expands support. Even the random baseline is not directly comparable in its current aggregate because its denominator includes goals the learned cell could not represent. I can record the coverage frontier; I cannot record a scaling result. The frontier was measured twice—once in the probe and once in the collected buffer—and both say that random reward-free data fails to contain the deep tree. That is durable because it changes how every later zero must be interpreted. The individual MC-FB success rates remain partial artifacts, not a finding, until a matched comparator and the missing axes exist.

record_finding()

{
  "title": "Random reward-free play does not support Craftax's deep achievement tree",
  "statement": "In Craftax-Classic-Symbolic, random-policy pretraining data covers only shallow achievements; deep-goal zero-shot failures under that data cannot distinguish representation capacity or representation family because the target states are absent from support.",
  "evidence": "A random-policy probe over 512 rollouts of 256 steps reached 9 of 22 achievements and reached none at depth 3 or deeper. The collected buffer over 1024 rollouts of 256 steps contained representatives for 8 achievements, all at depth 2 or shallower.",
  "caveats": "This establishes a support failure for random data, not that exploration coverage is the global bottleneck. The RND arm was not completed, and only one recurrent MC-FB d=64 cell was available."
}

toolrecord_finding

{"finding_id":1,"recorded":true,"scope":"random-data support frontier only"}

assistant

The run has been stopped by the user before the steered priority could produce the missing comparison. I am preserving the distinction between cancellation and a negative scientific result: there is no completed RND arm, no memory result, and no representation comparison to summarize. The stop converts this into an unfinished investigation with one useful precondition measurement, not a failed successor-measure study.

stop_experiment()

{
  "run_id": "R-f2b4",
  "reason": "User stopped the run before the main sweep completed. Preserve the coverage probe, corrected evaluator, smoke-test artifacts, random baseline, and completed MC-FB cell; do not mark missing arms as negative results."
}

toolstop_experiment

{"run_id":"R-f2b4","status":"stopped","stopped_by":"user","retained_artifacts":["coverage probe","reward-free random dataset coverage","corrected evaluator","MC-FB/HILP/GCBC smoke checks","random baseline","one recurrent MC-FB cell"],"scientific_status":"incomplete"}

assistant

Where this leaves the question. Established: random reward-free play is an invalid substrate for adjudicating deep zero-shot control in this environment because it does not reach the deep achievement states. The evaluator was corrected before the main read, and all three method paths passed end-to-end smoke tests. Not established: whether MC-FB or HILP scales with representation dimension, whether RND expands the supported goal frontier, whether recurrence matters more at depth, whether hierarchical successor measures compose, or whether the representation transfers to held-out worlds. The source has designs for those questions, not answers. The next runnable decision is narrower than the original ambition: first complete a strong-coverage dataset and verify that its achievement support actually moves beyond the random frontier. Only then run the fixed reference cell with recurrence on and off and compare methods on the identical supported-goal set. If stronger exploration does not move support, representation sweeps and hierarchical companions are premature. If it does, capacity and memory become interpretable. I would still not launch the composition or procedural-generalization companions until that reference cell is characterized.

labels

successful_investigation:false

primary_conclusion:“The session established only that random reward-free play has inadequate support for Craftax's deep achievement tree, so deep-goal zeros under that data cannot adjudicate successor-measure capacity or method quality. The sole launched run stopped after smoke c…”

thin_empirical_evidence:true

proposal_heavy_session:true

negative_result_reported:true

negative_result_diagnosed_mechanistically:true

implementation_gate_required_before_interpretation:true

support_failure_separated_from_representation_failure:true

headline_method_comparison_rejected:true

direct_minecraft_experiment_deferred:true

unlaunched_followups_not_reported_as_results:true

scope_cut_to_next_decision:true

unsupported_claims_present:false

all_quantitative_claims_traceable_to_tool_results:true

RL For Novel LLM Capabilities

Synthetic SFT

provenance

tools

messages

labels

steering_moves