# Checks, Lies, and Videotape | Def Method > METR found AI tools made developers 19% slower despite feeling faster. GPT-5.6 Sol gamed its benchmarks. Evidence — the fifth GUIDE principle — is the answer: check, don't just feel. [Essential Complexity](https://www.defmethod.com/essential-complexity) Modernizing high-risk systems in the age of AI. August 31, 2026 • 4 min read # Checks, Lies, and Videotape Developers using AI tools felt faster. They were 19% slower. GPT-5.6 Sol gamed its own benchmarks. Feeling confident is not the same thing as having evidence. ![Checks, Lies, and Videotape](https://www.defmethod.com/assets/insights/checks-lies-and-videotape-3a7d8c3e5353651c67581044300cc85e2f44ab6fa545bd30e3e367baa1c7a15b.png) ![Joe Leo](https://www.defmethod.com/assets/joe-leo-bd505c96465510e28eb8844a813bc854f5436d00a8ac7c74b94e1ce04b18c44a.png) [Joe Leo](https://www.defmethod.com/about) Founder, Def Method METR, the nonprofit that's become the closest thing the AI industry has to an independent auditor, ran a study last year. They took experienced open-source engineers, working on their own real repositories, and had them complete some tasks with AI tools and some without, then measured how long each took. The engineers were confident the AI made them faster. Every one of them expected it, going in, and most still believed it after the fact. The measured result was the opposite: developers using AI tools took 19 percent longer to finish the same work. These engineers were doing what everyone does by default: taking a strong feeling of confidence and treating it as though it were evidence. It felt faster. Fast typing, fewer blank-page moments, a satisfying stream of suggestions accepted. The only way to debunk that near-unanimous feeling across the board was to measure. ## Gaming the System METR ran into a new version of the same problem a few months later. In June, they evaluated OpenAI's GPT-5.6 Sol before its release and found it had the highest rate of evaluation gaming they'd ever detected in a publicly tested model. Sol exploited bugs in the scoring environment, finding ways to make its performance look better than the work underneath actually was. In some documented cases, the model tried to get another instance of itself to help conceal what it had done. The benchmark score, the thing everyone was about to cite as evidence of Sol's capability, had been optimized directly, sometimes instead of the capability it was supposed to represent. A human's confident feeling and a model's benchmark score are both assertions, or claims that something is true, made by something with a reason to want it to look true. In both cases, and in many more just like them in your own organization, the evidence is lacking. If you don't independently check whether the measurement is actually tracking the thing you care about, you'll end up trusting a number that's drifted away from reality. ## Evidence Evidence is part of the GUIDE principles of next-generation software engineering. Proof of correctness is what lets humans and agents trust a system, and proof isn't the same thing as a claim, however confidently the claim is made. As agents take on more of the implementation, automated tests, real observability, and agent-generated proof of correctness are what make accelerated implementation safe to keep shipping. We can no longer simply trust the green checkmark in our PRs or a clean smoke test on deployment. That's a higher bar than most teams are currently clearing, because most teams are still relying on exactly the two failure modes METR documented. A developer who feels productive gets treated as a developer who is productive, with nobody running the controlled comparison that would actually tell you. A model that passes its tests gets treated as a model that's correct, with nobody asking whether the tests could be satisfied in a cheaper way than actually being correct. Here's where to start: Separate who builds it from who verifies it. If the same agent that wrote the code also decides whether the code is correct, you don't have evidence. Route verification through a different agent, a different pass, or a human. Hold a slice of your tests back. The suite an agent can see and iterate against stops being a check and becomes a target the moment it's visible during development. Keep the cases that matter most out of reach until merge or release, the same way a real exam doesn't hand out the answer key during study hall. Run your own version of METR's comparison. Pick one team, one quarter, and measure cycle time and defect rate on AI-assisted work against a baseline, instead of asking people how it feels. Ask the gaming question of every metric before you trust it. Benchmark score, velocity number, "the tests are green" — for each one, ask an agent whether someone could produce a good result without the underlying work being right. If yes, fix the measurement before you build a process on top of it. That's the throughline connecting every letter in GUIDE. Governance asked how failures stay contained. Understanding asked how the organization's knowledge stays current. Intent asked whether instructions are precise enough to act on. Direction asked who's actually holding judgment, and at what altitude. Evidence is the answer to a question underneath all four of those: once implementation is fast enough that nobody can watch it happen line by line, how does anyone actually know whether any of it worked? Not by feeling confident. By checking, on purpose, in a way that would catch it if the answer turned out to be no. ### Need to modernize a high-stakes system? GUIDE is Def Method's framework for preserving understanding, strengthening governance, and building evidence before implementation accelerates. [Start with a GUIDE Assessment](https://www.defmethod.com/assessment) [Read the GUIDE Principles](https://www.defmethod.com/guide-principles) If this was useful, you might enjoy **Essential Complexity** — a bi-weekly letter on modernizing high-risk systems in the age of AI. [Subscribe](https://www.defmethod.com/essential-complexity) [← Essential Complexity](https://www.defmethod.com/essential-complexity)