The problem
Vault is a benchmark for comparing one agent with a supervised team against the same game and the same specification. I was interested in the part that a finished screenshot cannot show: whether the process produced evidence that survived inspection and whether the assembled build still matched the thing the reviewers thought they had verified.
The team run had separate seats and a larger process around them. The solo run had a much smaller history. That difference made it possible to compare not just output, but the records, identities, files, and verification work around the output.
Architecture
The benchmark is the Vault game repository plus the crosstalk workflow around it. In the team run, each seat worked through a per-seat branch and a pull request per slice, with no force-push process. The team also kept a peer-authored round-two bug list and a VERIFICATION.md evidence table for the assembled build.
The .verify/ directory held small watcher tools for playthrough, repair, queue, raid, reload, prop checks, and assignment. The game had generated textured assets for 18 of 19 room kinds, with zero regenerations. The solo comparison had the same BUGS.md and JOB.md baseline but did not carry the team's full shipping and verification document set.
Try it
The team build, deployed as a static site. Desktop mouse and keyboard; it is a game, so give it a few seconds to load.
Vault · team build
Three.js build, loads on demand
Decisions
Give each team seat a branch and slice PR
The SHIPPING process assigns each seat a branch and keeps one pull request per slice, with no force-push. That leaves a reviewable path from work to assembled build and keeps a correction attributable to a slice. Alternative considered: a shared branch and direct edits, which would reduce ceremony but make the evidence trail harder to separate.
Verify the assembled build with an evidence table
I used VERIFICATION.md to record evidence against the assembled build because a screenshot can hide the exact condition being checked. The elevator-waiter screenshot became a near-false-alarm: eyeballing it suggested a problem before the queue behavior was understood. Alternative considered: visual inspection alone.
Put repeatable checks in .verify/
The watcher scripts make playthrough, repair, queue, raid, reload, prop, and assignment checks repeatable. They also make it possible to inspect what a probe actually exercised. Alternative considered: rerun the game manually and rely on a reviewer to remember which path was covered.
Security / reliability
The team shipped a process record in SHIPPING.md, the assembled-build evidence table in VERIFICATION.md, and watcher scripts under .verify/. Those files matter because the benchmark is about the reliability of the verification loop as much as the game. A watcher can say that a path appeared healthy, but the probe still has to be challenged with a mutation and checked against the exact build under review.
Numbers
What went wrong
The verification probe passed the wrong thing
The clearest failure was the auto-assign check. Six out of six looked good until peer-4 introduced an if (true) return OK mutation. The mutated check then produced 0/8, so the verdict was MEANINGLESS. The build hashes also differed: 3a1e8b39 and 76be1636. A passing probe on one build was not evidence about the other.
The queue story was less dramatic but just as useful. The elevator-waiter screenshot was a near-false-alarm, and the first queue test proved nothing because the car holds four. B3 was withdrawn with the honest conclusion: my claim, and it does not survive measurement.
What I'd change
I would make mutation checks and build identity part of the normal verification record, not an investigative step after a result feels suspicious. I would also write the test's capacity and preconditions beside the screenshot or watcher output. That would make a green result easier to trust and a false alarm easier to dismiss.
Links
The team's repository is linked above as "vault-team on GitHub."