Live skill evaluations

embraion eval skills run --suite evals/skills/code-organization.json --host codex --route-class substantial --attempts 2 --output build/skill-evals.json runs actual native host sessions. It requires an installed, authenticated host CLI. The command does not install agents, skills, or dependencies.

Three hosts are supported:

• codex runs codex exec and installs suite skills under .agents/skills/<id>/.
• claude-code runs claude -p with stream-json output, loads only project settings and no MCP servers, accepts file edits inside the temporary project, and denies anything that would ask for permission. Suite skills go under .claude/skills/<id>/. Effort accepts low, medium, high, xhigh, and max.
• portable runs any agent CLI given with --host-command as a JSON argument vector, for example --host-command '["my-agent", "run"]'. The prompt arrives on stdin and the process runs in the temporary project, so a relative executable path resolves there; give an absolute path or a name on PATH. Suite skills go under --skill-dir (default .agents/skills). Only the exit code and the resulting files are observed, so tool calls, tokens, and trigger evidence stay unverified, and --model and --effort are refused; set them in the command.

--host-binary selects the CLI for codex or claude-code when it is not on PATH under its usual name; the portable host rejects it. On every host, more than 2 MB of standard output or error ends the session with output-limit. The report's evidence-kind names the host (live-codex-subprocess, live-claude-code-subprocess, live-portable-subprocess).

The suite schema is schemas/skill-eval.schema.json. Each case has a prompt, fixture directory, polarity (positive or negative), and declarative file checks. A suite lists one or more skills. Each skill has an ID and a candidate directory relative to the EmbrAIon root, with SKILL.md inside. Optional previous directories must be supplied for every skill in a composed suite. Paths, symlinks, source size, and duplicate IDs are checked before execution. The runner does not execute grader scripts from suites.

For every case and attempt, the runner copies the same fixture into a new temporary project. It runs a baseline without the suite skills and a candidate with the selected skill directories in the host's skill directory; when configured, it also runs a previous version. The source fixtures and skills are checked for mutation. Attempts are fresh sessions. Pair the same case and attempt across variants when examining results. The report includes suite, case, fixture, and skill SHA-256 identities so changed inputs are visible. It omits raw prompts, source text, model messages, and tool output. Temporary execution directories are removed at the end of each run.

behavior-passed requires a successful host completion and every declarative check to pass. The report also records host status, bounded elapsed time, observed tool call count, and token usage when the host emits it. Token counts follow each host's own accounting, for example of cached input, so compare them only between runs on the same host. A failed or malformed host run cannot pass behavior even if files happen to match. trigger-evidence is separate: read-observed means a host event showed a successful read of that skill's SKILL.md (a Codex command, or a Claude Code Skill or Read tool call whose result was not an error); unverified means this narrow event trace did not prove a read. It does not prove the skill was never loaded, and a read alone does not prove the skill caused the behavior. Negative cases test whether unrelated tasks stay within scope; their trigger absence remains unverified when host instrumentation does not expose loading.

Compare baseline, candidate, and previous results by case and repeated attempts. A candidate success alone does not establish lift; baseline may solve the task too. The sample suite covers English and Russian class additions and unrelated documentation edits. Fixtures are deliberately small. Extend suites with representative project constraints and failure modes before making broad claims. Results are sensitive to model, effort, host version, ambient instructions, and sampling variation. The runner selects the requested host settings but does not verify the host's internal model selection. Keep normal privacy, permission, routing, and security gates in force before sending suite prompts or fixtures to a provider.

The CLI resolves the project worker route before launching sessions. Use --path to select the project, --route-class for the task class and --data for its classification. Optional --model and --effort must match a configured route; only host-default resolution accepts requested settings. Routes with unsupported provider or additional options fail before launch.

Maintainer probe on 2026-10-03

The 0.20 runner was invoked with the English/Russian sample suite and two attempts. Its 16 native subprocesses reported host failure before any observed tool execution because this managed environment initially denied CLI state writes. A separate native probe with the required runtime write access reached the provider, which rejected the configured authentication token with HTTP 401. No real task completed, so skill quality, triggering and baseline lift remain unverified. Fake-host unit tests cover the runner mechanism; they are not behavioral improvement evidence. Repeat the live suite from an authenticated local Codex host before claiming semantic results.

Maintainer probe on 2026-10-06

The sample suite ran with one attempt per case on Claude Code 2.1.290 at its default model, once with --host claude-code and once with --host portable around the same CLI (--skill-dir .claude/skills). All 16 sessions completed and passed their file checks, so baseline and candidate tied on every case: the sample tasks are easy enough that the base model solves them without the skill. With claude-code, one candidate session (the Russian class addition) showed a successful skill load; the portable run cannot observe loading. This proves the runner works on both hosts, not that the skill improves behavior.