SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills

Large language model (LLM) agents accumulate rich episodic trajectories while solving real-world tasks, but it remains unclear whether such experience can be distilled into reusable procedural skills. We introduce SkillEvolBench, a diagnostic benchmark for evaluating this step from experience reuse to skill formation. It contains 180 tasks across six real-world agent environments, organized into role-conditioned task families with shared latent procedures. Agents learn from acquisition tasks, update an external skill library using compacted trajectories and verifier feedback, and then face frozen deployment tasks testing transfer under context shift, adversarial shortcuts, and composition. By comparing self-generated and curated-start skill evolution against no-skill and raw-trajectory controls, SkillEvolBench separates procedural abstraction from base capability, curated prior knowledge, and direct reuse of episodic traces.

Across ten model configurations and three agent harnesses, we find that current agents often adapt locally but rarely form robust reusable skills. Skill-based conditions can improve acquisition or replay, and individual models sometimes gain on specific deployment axes. However, these gains are unstable under context shift, adversarial shortcuts, and composition. Raw-trajectory reuse frequently outperforms distilled skills, suggesting that current abstraction procedures discard contextual and procedural cues that remain useful for future tasks. More frequent skill updating is also not monotonically beneficial, as additional revisions can improve coverage while introducing episode-specific drift. These findings highlight SkillEvolBench as a testbed for measuring when one-off experience becomes durable procedural knowledge.

Unlike static question-answering systems, practical agents interact with external environments over multi-step trajectories by reasoning, calling tools, inspecting files, executing code, editing artifacts, and observing feedback. Each task attempt therefore records how the agent acted, where it recovered, where it failed, and which checks exposed the failure. Prior experience-reuse methods show that these traces contain useful evidence, but reusing an episode is not the same as extracting a procedure. A trajectory records what happened once and often entangles transferable decisions with fixture-specific details, failed hypotheses, and mistakes.

SkillEvolBench targets the missing step between episodic memory and procedural reuse. If curated skills demonstrate that procedural knowledge can help agents, and trajectory-reuse methods demonstrate that episodes contain task-solving evidence, the central question is whether agents can convert noisy one-off experience into explicit skills that state what to do again, when to apply it, and what to verify along the way.

The benchmark spans common forms of agent work: code modification, API orchestration, data processing, document transformation, research synthesis, and communication operations. A task family corresponds to a recurring procedural capability rather than a topic label, so families are related enough for experience to matter but different enough to separate procedural learning from memorization.

Each family is instantiated through six roles. The canonical task presents the base procedure, the enriched task exposes a missing sub-capability, and the variant task changes the surface form while preserving the same procedure. The context-shift task embeds the skill need in a broader request, the adversarial task introduces shortcut solutions that can pass shallow checks, and the composition task requires the target skill to interact with other skills. This progression tests transfer, implicit invocation, shortcut resistance, and modular composition within the same family.

Acquisition uses the canonical, enriched, and variant roles. All acquisition tasks in an environment are completed before deployment begins. The active library is scoped to the environment: skills learned from earlier families may be visible to later families in the same environment, but skills never transfer across environments. Each acquisition attempt yields a compacted trajectory summary from harness-recorded artifacts such as instructions, file accesses, tool calls, commands, edits, generated outputs, tests, and final responses; verifier feedback includes outcome results, process checks, rewards, and diagnostics.

Skill authoring is family-local. Although the task-solving agent may read the environment-level library, the Skill Author receives only same-family skills and same-family acquisition history. After acquisition, the library is frozen: deployment tasks may read and apply accumulated skills but may not create, revise, retire, or otherwise modify them. Replay then reruns the original acquisition tasks using the final frozen library, providing a within-environment counterfactual for whether accumulated knowledge helps the tasks it was built from.

The experimental suite uses three agent harnesses under the same benchmark protocol: Claude Code, Codex CLI, and Gemini CLI. Claude Code is evaluated with Opus 4.6, Opus 4.5, Sonnet 4.6, and Sonnet 4.5; Codex CLI is evaluated with GPT-5.4, GPT-5.3-Codex, and GPT-5.2-Codex; Gemini CLI is evaluated with Gemini 3.1 Pro, Gemini 3 Flash, and Gemini 2.5 Pro. Success is measured at acquisition, replay, and frozen deployment, with deployment further decomposed into context shift, adversarial robustness, and composition.

The variants below distinguish several possible sources of improvement. NO-SKILL measures base agent capability without persistent memory. RAW-TRAJECTORY retrieves compacted acquisition traces without inducing skills, testing direct episodic reuse. Curated-start variants measure whether a useful but incomplete human-written skill can be refined from experience. Self-generated variants measure whether agents can induce skill artifacts from their own execution evidence and verifier feedback.