Essential Complexity

Modernizing high-risk systems in the age of AI.

• 4 min read

The Rise and Rise Again of Property-Based Testing

numpy.random.wald had been producing negative values for years in one of the most downloaded scientific libraries in the world. Nobody had written the test that would have caught it. An AI agent found it in minutes.

The Rise and Rise Again of Property-Based Testing
Joe Leo
Joe Leo

Founder, Def Method

numpy.random.wald draws random numbers from a distribution that cannot produce a negative value. It's a simple, useful function, and anyone who imported it depended on those values being non-negative. But there was a problem — numpy.random.wald did produce negative values!

The bug had been there for years in one of the most downloaded scientific libraries in the world. Nobody had written the test that would have caught it, because the test is almost too obvious to write: draw a number, check that it's positive. The fix, once maintainers landed it, reduced relative error by roughly ten orders of magnitude.

It was found by an agent that had been asked to come up with rules the code should obey, and then to try to break them.

Oh, Inverted World

The technique is called property-based testing (PBT), and Koen Claessen and John Hughes introduced it with QuickCheck in 2000. Instead of writing examples, you state a rule that has to hold and let a generator throw inputs at it until something violates it. A Wald distribution never returns a negative. Encode anything and decode it and you get the original back. Sort a list twice and nothing changes.

Tsong Yueh Chen had described a related idea in 1998, metamorphic testing, for the harder case where nobody can say what the right answer is. Take a search engine. Nobody can state the correct ranking for the query "running shoes," so there is no expected output to write down. But narrow the query to "running shoes under $100" and the result set cannot get larger. That relation holds no matter what the right answer is, and when it breaks you have found a bug.

Both ideas are stronger than example-based testing, and neither caught on broadly, for reasons we now have good data about. Harrison Goldstein, Joseph Cutler, Benjamin Pierce and Andrew Head at Penn, with Daniel Dickstein of Jane Street, interviewed thirty developers at Jane Street for an hour each and published the results at ICSE in 2024. Jane Street uses the technique heavily, so these were people who do it for a living. Their obstacles were almost entirely about labor. Sixteen of thirty said writing the specification slowed them down. Handwritten generators were described as tedious or high-effort. Shrinkers, which reduce a failing case to its smallest form, were worse: one participant said simply, "I hate writing shrinkers." Seventeen of thirty wanted help generating inputs that satisfy preconditions.

The Blueprint

The work those developers found tedious is work a model does well.

In January, researchers at Anthropic and Northeastern published their work on implementing PBT with AI models. They built an agent that reads a module and its documentation, proposes properties, writes Hypothesis tests, runs them, and then reflects on whether a failure is a real bug or a badly written test. They pointed it at more than a hundred popular PyPI packages.

The precision was mixed. Opus 4.1 produced 984 bug reports; of fifty reviewed by hand, 56% were valid and 32% were worth reporting. Filtering helped a great deal — among top-scoring reports, 86% were valid. But the bugs the study found, including the Wald example above, were real. That fix was merged. So was a function in AWS's Lambda Powertools that returned the same chunk over and over because an iterator was never advanced, and a hashing function in an AWS CloudFormation plugin that returned an identical value for every list because it called an in-place sort that returns nothing.

The research wave is broader than one lab. In April, a group proposed PropGen, which infers what a mobile app does from its interface, synthesizes properties from the observed behavior, and refines the imprecise ones. Across twelve Android apps it generated 985 properties, 912 judged valid, and surfaced 25 previously unknown bugs.

And we can, of course, follow the money. In December, Jane Street led a $105 million Series A in Antithesis, which sells simulation testing built on the same premise. Their engineer Doug Patti noted that leading an early round is rare for the firm. Antithesis counts MongoDB, Ramp and Ethereum among its customers. Ramp's VP of Engineering, Nikolay Koblov, put the problem in a sentence: "The most dangerous bugs are the ones we don't even know to look for."

Dear Science

The Anthropic team also found a bug in python-dateutil. Its easter() function returned dates that weren't Sundays for certain years under the Julian calendar. A reasonable property, clearly violated. The maintainers marked it invalid because behavior was intended.

A model can identify an anomaly but it cannot determine a defect, because that question is answered by intent. Intent, the third principle of GUIDE, resides with the people who built the library. The researchers say as much: deriving properties from code with subtle or implicit semantics remains hard, and maintainers are the final arbiter. That has an implication for production software, which is that if the Intent lives in someone's head it renders PBT much less effective. Intent must be codified.

Going back to the Penn interviews: that mechanical work is now largely automatable. But the most common failure those developers named was "not knowing what properties to test," and no amount of generation capacity touches that. The obstacles that disappeared were mechanical. The one that remains is about understanding the system. An advantage will accrue to engineering teams that build Understanding (the second GUIDE principle) and Intent alongside each other. A team that holds a theory of its system can now state its properties and check them at a cost that would have been prohibitive two years ago. One payoff is room to move. A property keeps holding while the code underneath it changes, so upgrades, refactoring and entire rewrites get easier. An agent can rework a module and the contracts either survive or they don't. A team without that boundary has one option when a large diff comes back, which is to read all of it. The other payoff grows as these tools get used more. More than half of that agent's hand-reviewed reports were not bugs. That ratio is manageable when someone can answer "is this behavior intended" in a few minutes, and it is not when the answer means tracking down the one person who remembers. The easter() case is the small version of that problem. Triage becomes the constraint, and triage can only be settled against intent somebody wrote down.

The cheapest place to begin is with the behaviors you would be upset to see silently changed. Write those down as properties before an agent touches the code.

Need to modernize a high-stakes system?

GUIDE is Def Method's framework for preserving understanding, strengthening governance, and building evidence before implementation accelerates.

If this was useful, you might enjoy Essential Complexity — a bi-weekly letter on modernizing high-risk systems in the age of AI.