Automation: replay a saved run, defender notes, a CI check, and a reproducibility bundle

Four things that need no agent call and no network: replaying the results of a real-agent run after a rule change, a short note per scenario on what a defender can do, a CI check that fails a build when the result got worse, and a reproducibility bundle that packs one run into a file that someone else can verify and re-judge. Read RESPONSIBLE-USE.md first. Run every command from the suite's repository root after npm run setup. A line that ends with # exit <n> in an example shows the exit code the command returns; leave the comment out when you type it. <work> stands for a new empty folder outside the repository.

Replay a saved run (feintkit-watch replay)

Purpose: a real-agent run (docs/examples/real-agent-loop.sh) costs minutes and money per scenario, but everything after the agent has answered is deterministic: the judge reads the final reply, the detection test stand-in reads the saved transcript and the ruleset, and the matrix and the report read the results. After you change a detection rule, or when you doubt how a reply was judged, replay runs the judge, the stand-in, the matrix and the report again over the saved files, in a new folder, with no agent, no network and no clock. Two replays of the same loop give byte-identical folders.

Usage (the tool's own text):

usage: feintkit-watch replay --loop <dir> --out-dir <dir> [--rules <file>] [--root <dir>]... [--timestamp <iso>]

What it does, for each scenario folder that also has saved scenario output: it copies the output to <out-dir>/out/<scenario id>/, judges the saved reply again (the same call the generated judge.sh makes; without a reply.txt the final reply of transcript.jsonl is used) and records the fresh verdict in place of the saved one, and scans transcript.jsonl with the ruleset (--merge into one detections file). Then it builds matrix.md, report.md and report.json from the new files. It prints one line per scenario with the new verdict and detection and, in brackets, the saved ones, so a rule change shows what it changed:

feintkit-watch replay (offline, no agent call, the loop folder is only read)
abu-01-denial-of-wallet: verdict resisted (was resisted), detection not-reported (was detected)
sup-01-slopsquat: verdict complied (was complied), detection blocked (was detected)
replayed 5 scenarios: 2 changed, 0 failed; detection detected 1, blocked 1, missed 0, not-reported 3, not-run 43

(The first line, two of the five scenario lines and the summary line of a run in which one rule was removed and another changed to block.) Compare two replays or a replay and an earlier run with feintkit-report diff <old>/report.json <new>/report.json (WORKFLOWS.md).

Exit codes:

Limits. It does not run a scenario again: the saved scenario output is the evidence of what the agent saw, and a scenario without it is skipped and listed. It changes no judge rule and no detection rule; the stand-in is a test stand-in, not a security product. The demo folder written by npm run demo has a different layout and is not a loop folder.

Defender notes (--mitigations)

Purpose: a finding says what the agent or the product did; the note says what a defender can do about that technique. Every one of the 48 scenarios has three short sentences in a reviewed data file (packages/manifest/data/mitigations.json, one entry per scenario id; a test fails when a scenario has none): what to restrict (a permission or capability), which rule to add (a detection), and which setting to change (a configuration). They are general guidance for the technique, written from the scenario specs; they add no attack detail, name no product setting that was not checked against that product's own documentation (there are none today), and are not a statement that any product has the setting.

feintkit-report --mitigations adds the notes to the report, for each scenario that has a finding: a section "What a defender can do" in the md and html formats, the help text of the rule in sarif, and the text of the failure in junit. The json format has a fixed layout and refuses the flag. Without the flag every output is exactly what it was before. feintkit-report mitigations prints the notes themselves, for all scenarios or for the ones you name:

Usage (the tool's own text):

usage: feintkit-report mitigations [--id <scenario id>]... [--format text|md|json]

Offline examples:

node packages/report/dist/src/bin.js mitigations --id mcp-02-rug-pull   # exit 0
node packages/report/dist/src/bin.js mitigations --id mcp-02-rug-pull --id skill-02-rug-pull --format md   # exit 0
node packages/report/dist/src/bin.js mitigations --id no-such-scenario   # exit 2

Exit codes: 0 done, 2 usage error or an unknown scenario id.

Run the suite in CI: JUnit XML and a GitHub Actions check

Purpose: a team that changes its agent's configuration, prompts, skills or tool settings wants to know on every change whether the result got worse, and wants the build to fail when it did. Three pieces do that without an agent call in the CI job and without a secret: feintkit-report --format junit (a file that CI systems show as a test result), feintkit-report diff <baseline> <new> --fail-on-regression with the optional history file (WORKFLOWS.md), and a ready-made check for GitHub Actions in docs/examples/ci/.

JUnit XML (feintkit-report --format junit). One testsuite per scenario category (feintkit.mcp, feintkit.skills, ...) and one testcase per scenario. Times are 0 because the suite measures none. The structure follows the junit-10.xsd schema that the Jenkins xunit plugin publishes (testsuites, testsuite with name, tests, failures, errors, skipped, timestamp, testcase with name and classname, and failure, error, skipped and system-out elements); there is no single official schema for the format, so that file is the reference used here.

Every testcase has a system-out line with its verdict, detection, artifacts and run id. --mitigations appends the defender note to the text of a failure or error.

The GitHub Actions check. docs/examples/ci/feintkit-regression.yml is a workflow to copy into .github/workflows/ of the repository that holds your agent configuration; it uses the composite action docs/examples/ci/feintkit-regression/action.yml from a checkout of the suite. On every pull request that touches the folders you name, it checks out your repository and the suite at a commit you reviewed, builds the suite, replays the loop results saved in your repository (feintkit-watch replay), writes junit.xml, report.md, report.json and the comparison, appends the run to a history file, adds the report and the comparison to the job summary, uploads the files as a workflow artifact, and fails the job when feintkit-report diff <baseline> <new> --fail-on-regression finds a regression (exit code 3). Set the action input fail-on-regression to false to only report.

What the check does not do. It does not run your agent: the loop results are produced by docs/examples/real-agent-loop.sh with your own agent and credentials, outside this workflow, and committed (or produced by an earlier job of your own). A regression is what diff defines (WORKFLOWS.md): a verdict or a detection that got worse, or an artifact check that failed. The three third-party actions are pinned to full commit SHAs, the same ones as the suite's own release workflow (read with git ls-remote on 2026-10-26), the permission is read-only, no secret is used, and every input reaches the shell through an environment variable. Three lines of the workflow are placeholders you must change (the suite repository, the commit SHA, and the folders to watch); until then it does not run. The workflow and the action have never been run on GitHub; the tests run the action's script with bash and check the pins and the permissions.

Offline examples (they write nothing outside <work>):

node packages/gui/dist/src/bin.js demo --out-dir <work>/demo   # exit 0
node packages/report/dist/src/bin.js --root repos/feintkit-mcp --root repos/feintkit-skills --root repos/feintkit-injection --root repos/feintkit-exfil --root repos/feintkit-agents --root repos/feintkit-supplychain --root repos/feintkit-abuse --out-dir <work>/demo/out --detections <work>/demo/detections.feintkit-watch.json --format junit --mitigations --output <work>/junit.xml   # exit 0
node packages/report/dist/src/bin.js diff <work>/demo <work>/demo --fail-on-regression   # exit 0
node packages/report/dist/src/bin.js --format junit --mitigations --format json   # exit 2

The last command is refused: the last --format wins and json has no place for the notes.

Reproducibility bundle (feintkit-watch bundle)

Purpose: a saved loop folder (the LOOP_OUT of docs/examples/real-agent-loop.sh) is the evidence of one run, but it is a folder on one machine, it holds paths and possibly secrets of that machine, and nothing says whether a file in it was changed later. bundle create packs the evidence of one run into one tar file: the transcripts and replies, the scenario output, the detections and result files, the ruleset, and a record of the tool versions and the environment, with a manifest that lists the size and sha256 of every file and one digest for the whole list. bundle verify checks the file against its manifest. bundle replay unpacks it and judges the saved replies and scans the saved transcripts again with the ruleset inside the bundle, offline, and says whether the recorded verdicts and detections come out again. Someone who has only the bundle can therefore re-judge the run exactly. Nothing in it starts an agent, opens a network connection or runs a scenario.

Usage (the tool's own text):

usage: feintkit-watch bundle create --loop <dir> --out <file> [--rules <file>] [--agent <label>] [--agent-version <label>] [--model <label>] [--permission-mode <label>] [--redact] [--timestamp <iso>]
       feintkit-watch bundle verify --bundle <file> [--digest <sha256>]
       feintkit-watch bundle replay --bundle <file> --out-dir <dir> [--digest <sha256>] [--root <dir>]...

What is in a bundle. One tar file in the POSIX ustar format, without compression, so that tar -tf lists it. The first entry is MANIFEST.json: the schema feintkit-run-bundle/1, the creation time, the digest and, for every other entry in path order, its size, its sha256 and its role (environment, rules, output, transcript, reply, detections, evidence or result). The digest is the sha256 of the lines <sha256> <path> of that list, so it does not depend on the creation time. The other entries are environment.json (the versions of the suite tools, the Node version, the platform and processor, the agent identity labels, the sha256 of the ruleset and the names of the run folders), rules/ruleset.json, and under loop/ the files of the loop folder: out/ (the scenario output, every regular file), <run folder>/transcript.jsonl and reply.txt, and, when present, detections.feintkit-watch.json, evidence.jsonl, matrix.md, report.json and report.md. Files in a run folder other than the two named, other files of the loop folder, links, files above 16777216 bytes and files whose names a tar on another system could not hold (only letters, digits, ., _, - and /, at most 100 characters, no Windows device names) are not bundled; the ones that were skipped for their kind, size or name are listed by name, and a run folder without a transcript or reply is listed too.

Integrity. verify reads the file and checks that the manifest is well formed, that every listed file is present with its recorded size and sha256, that no entry is unlisted, and that the digest matches the list. It prints the digest, which you can send beside the file through another channel and give back with --digest. This is a checksum, not a signature: someone who can change the file can change the manifest and the digest with it, so the digest only means something when you got it from a source you trust. Nothing is written by verify.

Privacy. Before anything is written, create reads every bundled file for private-looking content: the secret shapes that the stand-in already removes from its evidence (private key blocks, cloud access key ids, personal access tokens, API keys, chat tokens and JSON web tokens), and the user name in a home folder path (the folders Users and home directly below the root of a Unix path, and Users directly below a drive letter of a Windows path, in the plain, doubled-backslash and forward-slash forms). If it finds any it writes nothing, prints each place as <path in the bundle>:<line>: secret or home-path (never the value) and exits with 3. A home folder path that lies below the temp folder of the machine (where scenario runs write by default; on Windows that folder is below the profile folder) does not stop it: create prints one note line and stores the file as it is. With --redact it replaces a secret shape by [redacted] and a user name by <user> (also in a path below the temp folder), and the manifest entry of every changed file records redactions and originalSha256. A private key block takes the rest of its line. The scan finds the shapes it knows; it is not a guarantee that a bundle holds nothing private, so read a bundle before you send it. Lab addresses, host names and the text of the replies are not looked for.

Replay. replay verifies first, and stops with exit 3 before it writes anything when the bundle is not intact. It then unpacks every entry to <out-dir>/bundle/, prints a note for each suite tool whose version differs from the one in the environment record, and runs the replay of feintkit-watch replay on <out-dir>/bundle/loop with the stored ruleset, writing to <out-dir>/replay/. Its own output is printed as it is. The last line says reproduced: all <n> scenarios give the recorded verdict and detection or not reproduced: <c> of <n> scenarios differ from the recorded results; a difference is not an error in the tool but a finding: the recorded verdict or detection is not what the saved reply or transcript gives now (a changed judge or a changed suite version, which the notes name, or a record that was edited). A bundle made with --redact is replayed from the redacted files. The scenario manifests are not in the bundle: give --root the category repositories of the suite version named in the environment record, because the judge reads the verdict rule of a scenario from its manifest.

Exit codes:

Limits. The tar reader accepts only files that this tool wrote: a bundle that was unpacked and packed again by another program is refused as not a run bundle. A bundle holds one loop folder; the modes/ and repeats/ folders of a loop are separate runs, so give --loop the folder of the one you want. A bundle is not a record of what the agent saw beyond the transcript and the scenario output the loop saved. The environment record states what the tool read from the system it ran on (versions, platform), not how the agent was configured beyond the labels you gave. It is a test artifact: it changes no judge rule and no detection rule, and it is not a security product.