5 docs tagged with "audit"

From manual ritual in Sentinel to canonized CLI command

The only file in the framework that lived in another language

A four-model audit cycle reviewed a Charter that added a new public method — the only method that made a feature work. Three of four auditors reported no critical or high findings. The feature had zero production callers, 3,376 passing tests, and did not work end to end. The defect was mechanically detectable with a single grep; one auditor ran that exact grep and read the result backwards. The post-mortem produced a one-rule addition to the audit prompt — and a more uncomfortable finding about the implementer's own verification, which could not produce a red result.

Bun's Rust rewrite ran AI adversarial review at a scale worth studying — one model family, split context windows, and 1,386,826 test assertions as the final judge. StrayMark's audits look different because the target is different, and the difference isn't stylistic. When a passing test suite can be the judge, you isolate reviewers by context. When the judge is a human judgment call, you diversify by model family — and 25 audit cycles of real data show exactly why, including one family that went quietly blind.

StrayMark's audit is only worth anything if independent model families converge on a finding by themselves — agreement is the signal. A run of releases hardened that guarantee against three ways to fake it. The freshest is the sharpest — a router CLI injects its own product name, so an auditor stamped its report "qwen-code" even after the operator had switched the backend model. The convergence math was being fed a lie about who did the work.