SE Radio 589: Zac Hatfield-Dodds on Property-Based Testing in Python

Zac Hatfield-Dodds, the Assurance Team Lead at Anthropic, speaks with host Gregory M. Kapfhammer about property-based testing techniques and how to use them in an open-source tool called Hypothesis. They discuss how to define properties for a Python function and implement a test case in Hypothesis. They also explore some of the advanced features in Hypothesis that can automatically generate a test case and perform fuzzing campaigns.

Zac Hatfield-Dodds, the Assurance Team Lead at Anthropic, speaks with host Gregory M. Kapfhammer about property-based testing techniques and how to use them in an open-source tool called Hypothesis. They discuss how to define properties for a Python function and implement a test case in Hypothesis. They also explore some of the advanced features in Hypothesis that can automatically generate a test case and perform fuzzing campaigns.

Show Notes

Related Episodes

SE Radio 256: Jay Fields on Working Effectively with Unit Tests

SE Radio 283: Alexander Tarlinder on Developer Testing

SE Radio 322: Bill Venners on Property Based Tests

SE Radio 474: Paul Butcher on Fuzz Testing

SE Radio 516: Brian Okken on Testing in Python with Pytest

Other References

• Zac Hatfield-Dodds’ Web site: Zac HD

• Documentation for Hypothesis: Welcome to Hypothesis — Hypothesis 6.88.3 documentation

• Web site for Hypothesis: Products – Hypothesis

• Documentation for the Hypothesis GhostWriter: Ghostwriting tests for you — Hypothesis 6.88.3 documentation

• GitHub repository for Hypothesis: GitHub – HypothesisWorks/hypothesis: Hypothesis is a powerful, flexible, and easy to use library for property-based testing.

• Documentation for HypoFuzz: HypoFuzz documentation — HypoFuzz docs

• Web site for HypoFuzz: HypoFuzz

Transcript

Transcript brought to you by IEEE Software magazine and IEEE Computer Society. This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.

Greg Kapfhammer 00:00:18 Welcome to Software Engineering Radio. I’m your host, Gregory Kapfhammer. Today’s guest is Zac Hatfield Dodds, an Australian researcher, software engineer, and open source maintainer. Zac leads the assurance team at anthropic and AI Safety and Research company building reliable, interpretable and steerable AI systems. With the goal of making testing easier for software developers, Zac maintains open source projects like Hypothesis, Pytest and HypoFuzz. In addition to being a speaker at conferences like PyCon, Zac was recognized as a fellow of the Python Software Foundation. Welcome to Software Engineering Radio, Zac.

Zac Hatfield-Dodds 00:01:02 It’s great to be with you Greg.

Greg Kapfhammer 00:01:04 Thanks for joining us today. We’re going to be talking about property-based testing and how developers can create property-based tests in Pytest. We’ll be learning how to do that in an open source tool called Hypothesis. Zac, before we dive into the technical details, can you give us a brief introduction to property-based testing and how it works with the Hypothesis tool?

Zac Hatfield-Dodds 00:01:27 Absolutely. So when I think about writing software tests and the kind of unit test style, maybe they’re testing small units, maybe they’re testing whole programs. There are basically two problems that you have to solve in order to have a software test. The first is that you need to work out what kind of input data you run, and the second is that you need to check in some way that your code didn’t do the wrong thing. In Python, that usually means you run a test function and the test is considered to fail if it raises an exception or in unit test if it calls a particular assert method with a false value. And the second part is coming up with the test inputs that actually elicit that behavior that might be buggy. And so property-based testing gives us a different attitude to what we check and a much richer set of tooling to help computers generate those inputs for us.

Greg Kapfhammer 00:02:18 So one of the things that I’m learning is that as a tester you have to think about both the inputs and the checks and it sounds like Hypothesis helps us to do that in an automated fashion. Can you share a success story associated with the use of property-based testing and the Hypothesis tool?

Zac Hatfield-Dodds 00:02:37 One of my favorite case studies is from a friend of mine who some years back was teaching a machine learning course. This was long enough ago that PyTorch was not yet installable on Windows. And so to help his students get started, he wanted to have an auto diff library that could be used for simple machine learning problems but was much easier to install. And he credits Hypothesis with making that possible. He says that writing an advanced and correct library that covered all of the edge cases would’ve been impossible for one person to do before the course started if he hadn’t had these powerful techniques for testing it.

Greg Kapfhammer 00:03:12 So it sounds like property-based testing and Hypothesis helps us to do things like narrow in on the edge cases in our program under test. I’d love to learn more about property-based testing later in the show, but in order to ensure that we set a firm foundation, can you give us a brief definition of a few software testing concepts? So to get started, for example, what is a test case and a test suite?

Zac Hatfield-Dodds 00:03:37 Great questions. So there are a couple of pieces here. When we say test suite, what we usually mean is all of the tests that we have. So if you have your CI system set up to run a bunch of tests on each pool request, for example, all of the stuff that gets run collectively is your test suite. A test function is then one particular functional method that’s part of your test suite. And a test case is one particular input. So you might have a parameterized test, sometimes people call them table driven tests where you write down many inputs and outputs. We would say that that is a single test function, a part of your test suite and that the table contains many test cases. The thing you are testing is usually called the system under test, sometimes function under test in hardware, people would say device under test.

Greg Kapfhammer 00:04:31 That makes a lot of sense. Can you tell us what’s the meaning of the term example based test case?

Zac Hatfield-Dodds 00:04:37 So an example based test is pretty much defined by contrast to property-based tests. A standard example based test is where you write down a particular input to your code. Then usually you call your code on that input and most often you then assert that the output you actually got is what you expected. And so because you are thinking up this single example for yourself, we often call that an example based test.

Greg Kapfhammer 00:05:03 Okay. So would you agree that many times when people use Python testing tools like unittest or Pytest, that example based testing is the standard approach to writing test cases?

Zac Hatfield-Dodds 00:05:17 Yeah, I do think it’s the most common approach to testing in this style.

[...]

Original source