PostHole
Compose
Login
SE Radio 685: Will Wilson on Deterministic Simulation Testing
In this episode, Will Wilson, CEO and co-founder of Antithesis, explores Deterministic Simulation Testing (DST) with host Sriram Panyam. Wilson was part of the pioneering team at FoundationDB that developed this revolutionary testing approach, which was later acquired by Apple in 2015. After seeing that even sophisticated organizations lacked robust testing for distributed systems, Wilson co-founded Antithesis in 2018 to make DST commercially available. Deterministic simulation testing runs software in a fully controlled, simulated environment in which all sources of non-determinism are eliminated or controlled. Unlike traditional testing or chaos engineering, DST operates in a separate environment from production, allowing for aggressive fault injection without risk to live systems. The key breakthrough is perfect reproducibility -- any bug found can be recreated exactly using the same random seed. Antithesis built "The Determinator," a custom deterministic hypervisor that simulates entire software stacks including virtual hardware, networking, and time. The system can compress years of stress testing into shorter timeframes by running simulations faster than wall-clock time. All external interfaces that could introduce non-determinism (network calls, disk I/O, system time) are mocked or controlled by the simulator. The approach has proven effective with major organizations including MongoDB, Palantir, and Ethereum. For Ethereum's critical "Merge" upgrade in 2022, Antithesis found and helped fix several serious bugs that could have been catastrophic for the live network. The platform typically finds bugs that traditional testing methods miss entirely -- such as those arising from rare race conditions, complex timing issues, and unexpected system interactions. This episode is sponsored by Monday Dev
In this episode, Will Wilson, CEO and co-founder of Antithesis, explores Deterministic Simulation Testing (DST) with host Sriram Panyam. Wilson was part of the pioneering team at FoundationDB that developed this revolutionary testing approach, which was later acquired by Apple in 2015. After seeing that even sophisticated organizations lacked robust testing for distributed systems, Wilson co-founded Antithesis in 2018 to make DST commercially available.
Deterministic simulation testing runs software in a fully controlled, simulated environment in which all sources of non-determinism are eliminated or controlled. Unlike traditional testing or chaos engineering, DST operates in a separate environment from production, allowing for aggressive fault injection without risk to live systems. The key breakthrough is perfect reproducibility — any bug found can be recreated exactly using the same random seed.
Antithesis built “The Determinator,” a custom deterministic hypervisor that simulates entire software stacks including virtual hardware, networking, and time. The system can compress years of stress testing into shorter timeframes by running simulations faster than wall-clock time. All external interfaces that could introduce non-determinism (network calls, disk I/O, system time) are mocked or controlled by the simulator.
The approach has proven effective with major organizations including MongoDB, Palantir, and Ethereum. For Ethereum’s critical “Merge” upgrade in 2022, Antithesis found and helped fix several serious bugs that could have been catastrophic for the live network. The platform typically finds bugs that traditional testing methods miss entirely — such as those arising from rare race conditions, complex timing issues, and unexpected system interactions.
Show Notes
Related Episodes
Related Resources
- Interview with a DST practitioner: Antithesis-Driven Testing
- Write-up on Ethereum testing: Testing the Ethereum Merge
- FoundationDB paper (PDF) – see section 4 and section 6.2
- 2015 talk at StrangeLoop: Testing Distributed Systems w/ Deterministic Simulation by Will Wilson
- A more recent talk: Testing a Single-Node, Single Threaded, Distributed System Written in 1985 by Will Wilson
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Sri Panyam 00:00:18 Hello and welcome to Software Engineering Radio. This is your host, Sri Panyam. Today I’m joined by Will Wilson, CEO and co-founder of Antithesis. Will started off in biotech and went on to work on distributed systems at FoundationDB, Apple and Google. And then he eventually went on to found Antithesis, an autonomous testing platform based on Deterministic Simulation Testing (DST). Will, welcome to the show.
Will Wilson 00:00:45 Thank you so much for having me.
Sri Panyam 00:00:47 For our listeners who may not be familiar with the concept, could you share what is DST or Deterministic Simulation Testing?
Will Wilson 00:00:54 Yeah, sure. So let’s start with the way that people normally test software. So the way you normally test software is first you write some code and then you try and think about what situations you want to cover with a test, what situations you want to make sure are going to work in real life, and then you write some tests to cover those situations. And then you occasionally run those tests. Maybe you do them on every CI build or maybe you do them on every PR, maybe you do them only when you’re about to release your software, but they cover those cases and you hope that when you deploy to production, everything is going to go okay. Now let me ask you a question. Do you believe if all your tests pass that everything is going to be okay?
Sri Panyam 00:01:35 I would like to think so.
Will Wilson 00:01:36 But do you actually?
Sri Panyam 00:01:38 Well, if anything in the last 20 years has shown me, no..
Will Wilson 00:01:42 Exactly. So why is that? Well, it’s because, I just said you are writing the tests that cover the cases that you thought were important to cover, but the things that are actually going to screw you in production are the situations that you did not think of ahead of time. They’re the things you did not think to cover. And so what that means is this conventional strategy to writing tests is very, very good for catching regressions. If there’s something that I know my software does correctly and I accidentally break it in a new version, if I have a test for it, that’s pretty good at catching that. But a test is very bad at finding out if my software actually works in the real world in all the unanticipated or unexpected situations that the real world will throw at me. So deterministic simulation testing is a very different strategy which tries to actually give us the confidence that if our test passed, things will go well in the real world.
Will Wilson 00:02:38 And the way it does that is basically by constructing a model of the real world, a whole system simulation where random requests come in and ask your software to do things and tasks are going to run in random orders and these requests are going to have slightly randomized parameters. And also the things that happen in the real world that can go wrong are also going to happen. So occasionally requests will fail, requests to downstream dependent systems will fail, network requests may get delayed or reordered, threads may randomly pause and run later. The garbage collector may just turn on and that whole node may slow down. All the stuff that causes people to get paged in the middle of the night and causes weird production behaviors that nobody can understand, we’re going to do all that too. And we’re going to do it at a vastly accelerated rate so that we cover many, many years or centuries worth of real-world badness in just a few seconds or minutes in our tests.
[...]
📄 fdb-paper.pdf