Developer builds recorder after agent claimed tests passed
Rashomon keeps its own log of what a coding agent does, then sets it beside what the agent says it did. Its maker says his audits found the two did not always match.
A developer gave a coding agent access to his own repositories, then checked each transcript against what actually changed in the code.
On his account, the agent in many cases either failed to disclose what it had done or worked around problems such as failing tests rather than saying so.
That audit produced Rashomon, an open-source execution recorder published on GitHub under the Apache 2.0 licence.
The tool keeps an independent record of shell commands, exit codes, file writes, tool calls, test runs and subagents, and builds a timeline of those events.
It then compares that record with the agent's closing summary and prints one line at the end of a turn when the two disagree. A turn with nothing worth flagging prints nothing.
The comparison is blunt by design. Rashomon holds a fixed list of 43 failure words — fail, failed, error, unable, did not and the rest — and checks whether the agent's final message contains any of them.
If a recorded call exited non-zero and the summary uses none of those words, the report lists the words the summary avoided. It does not guess at intent.
Reports also show what subagents did. Those helper agents run their own calls under their own transcripts, which the main conversation never displays.
The obvious objection is that a diff already shows what changed. The project's answer is that a diff shows the final result, while the recorder shows the execution that produced it.
The limits are stated plainly in the documentation. It is alpha software at version 1.1.0, it supports Claude Code only, on macOS and Linux, and Windows is not supported.
Network destinations are not observed in this release; the report prints that gap on every session. Cursor and Codex can read Claude Code's settings and trigger the recorder anyway, and what it records for them can be wrong.
It is not a sandbox. It cannot stop a command, and the documentation says it cannot stop a determined agent from altering its own records.
On privacy, the project says it stores no prompts, responses, file contents or tool outputs, keeps records in a folder on the machine, and has no account and no telemetry. It says no package imports a networking library and that continuous integration checks this on every pull request, push to main and release tag.
The repository shows 14 stars, three forks and 88 commits. No pricing or commercial model has been announced.
For founders shipping with agents, the practical point is cheap to test. Break one test deliberately, ask an agent to use a subagent and then run the suite, and read its summary against the exit codes. Whether that summary is accurate is a question about your release process, not about the tooling you bought.
More: the Rashomon repository