PostHole
Compose
Login
SE Radio 622: Wolf Vollprecht on Python Tooling in Rust
Wolf Vollprecht, the CEO and founder of Prefix.dev, speaks with host Gregory M. Kapfhammer about how to implement Python tools, such as package managers, in the Rust programming language. They discuss the challenges associated with building Python infrastructure tooling in Python and explore how using the Rust programming language addresses these concerns. They also explore the implementation details of Rust-based tooling for the Python ecosystem, focusing on the cross-platform Pixi package management tool, which enables developers to easily and efficiently install libraries and applications in a reproducible fashion. Brought to you by IEEE Computer Society and IEEE Software magazine.
Wolf Vollprecht, the CEO and founder of Prefix.dev, speaks with host Gregory M. Kapfhammer about how to implement Python tools, such as package managers, in the Rust programming language. They discuss the challenges associated with building Python infrastructure tooling in Python and explore how using the Rust programming language addresses these concerns. They also explore the implementation details of Rust-based tooling for the Python ecosystem, focusing on the cross-platform Pixi package management tool, which enables developers to easily and efficiently install libraries and applications in a reproducible fashion. Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episodes
- 562: Bastian Gruber on Rust Web Development
- 490: Tim McNamara on Rust 2021 Edition
- 279: Florian Gilcher on Rust
- 581: Zach Lloyd on Terminal Emulators
- 589: Zac Hatfield-Dodds on Property-Based Testing in Python
- 431: Ken Youens-Clark on Learning Python
Other References
Transcript
Transcript brought to you by IEEE Software magazine and IEEE Computer Society. This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number.
Gregory M. Kapfhammer 00:00:18 Welcome to Software Engineering Radio. I’m your host Gregory Kapfhammer. Today’s guest is Wolf Vollprecht, the CEO and founder of Prefix.dev. He is the creator of both the Mamba Package manager implemented in C++ and the Pixi package manager implemented in Rust. Wolf is also the creator of numerous packages on Conde Forge, a core member of the Conde Forge team and the founder of the Robo Stack Project. Welcome to Software Engineering Radio, Wolf.
Wolf Vollprecht 00:00:49 Thanks for having me. I’m very excited to be here.
Gregory M. Kapfhammer 00:00:52 I’m glad you’re here. Today during our episode we’re going to be talking about Rust based tooling for the Python programming language. To keep our conversation concrete, we’re going to focus on the Pixi package manager. To get our conversation started, I have a two-part question. The first part is, what is the Pixi package manager and then secondly Wolf, why did you choose to implement it in Rust?
Wolf Vollprecht 00:01:17 Excellent questions. So for Pixi, we are actually starting to also call it the workflow manager. So it’s not just a package manager. And Pixi is a project that we’ve recently started to kind of kick off the next generation package management for the Conda ecosystem. And we are sitting on top of the or standing on top of the shoulders of the entire existing Conda ecosystem. And so all the packages that already exist are compatible with Pixi. And Pixi gives you a very modern workflow to work with projects, using these operating system agnostic packages in a way. So you can easily manage your projects on Windows, Mac OS and Linux using Pixi as a package manager. And it’s also at the same time through the roots of our like Conda heritage in a way, we have a very tight integration with the Python ecosystem as well. And yeah, Pixi basically lets you first of all manage your packages and dependencies, then create some cross-platform tasks that you can easily execute on any computer with the idea that all you have to do is get clone my project and then Pixi run start. And that will do all the magic that’s necessary to kind of get your project up and running on that computer that you are on.
Gregory M. Kapfhammer 00:02:32 That sounds really interesting, Wolf. We’re going to dive into the specific details of the Pixi workflow management system. Before we go into the details, can you tell us why did you choose to implement it in Rust?
Wolf Vollprecht 00:02:44 Yeah, for sure. So we actually started out with our low-level libraries and they’re called Redler and Redler implements their Rust crates, which is just a Rust word for libraries I think. And the RedLab project essentially implements everything you need to deal with Conda packages. So downloading what’s called repo data to installing the packages into a target environment and doing everything in between. It’s implemented in that low level library and this library is actually used across all of our projects. So on our website Prefix.dev, we use Redler in the background on our tool to build Conda packages. We use Redler with the tool is called Redler Built and Pixi. And so the first motivation for like using Rust here was that we could easily use it also in our server backend because before I was implemented or I was developing a lot on Mamba and Mamba has a C++ sort of library in the back and that was not easy to use in a server environment because of the memory safety issues of C++. And actually there’s no such an extensive ecosystem of like server libraries in C++ and it’s much easier to kind of do that with Rust in my experience.
Gregory M. Kapfhammer 00:04:05 Alright, thank you for that response. We’re going to start talking a little bit about some of the Python tools and the Python ecosystem and then later we’re going to investigate the specifics of how you built your systems using Rust. So let’s start with some of the basics of Python package management. There’s a couple tools that I think we should discuss. I anticipate some of our listeners will know about them, but Wolf if you could give an intro, what are the tools that are called PyPI and Pipx and how do developers use those?
Wolf Vollprecht 00:04:35 Yeah, so Pip is sort of the, I don’t know, grandfather is maybe the wrong word, but one of the basic package managers in the PyPI ecosystem. And the PyPI ecosystem again is something else than the Conda ecosystem. The PyPI stands for Python Package Index and its basically community-managed website that hosts all of the Python packages more or less in existence. And it’s very easy to just upload and self-publish your own package to the Python package index. It’s very widely used by Python developers and Pip is sort of the premier tool, like the community standard tool I would say, to install these packages. But it’s also, well I don’t want to say simple is the wrong word, but it implements a very bare bones kind of experience. It just basically fetches packages and installs them while other package managers maybe do some more things like lock files. But we’ll talk about that later, I guess. And then Pipx is a tool to take to make packages globally available on your CLI so that you can Pipx cowsay or something like this — and cowsay is just a fun executable — and when you install that with PipX it will be available on your comment line and you can just use it as if you would have installed it with apt get or something like this systemwide.
Gregory M. Kapfhammer 00:05:59 Yeah, thanks for those responses. Let’s talk now about some perhaps slightly more modern Python package management tools. The two that come to my mind are PipM and Poetry. Can you talk a little bit about how those tools work?
Wolf Vollprecht 00:06:12 So the honest answers that I haven’t extensively used either of those two tools .
Gregory M. Kapfhammer 00:06:18 Okay, that’s fine.
[...]