- https://hugovk.github.io/free-threaded-wheels/ is looking pretty healthy - 130 of the 360 most downloaded C extension PyPI packages are now free-threaded Python compatible, up from 92 on 15th August https://web.archive.org/web/20250815071755/https://hugovk.gi...
I was curious as to how that site works - it has a build script at https://github.com/hugovk/free-threaded-wheels/blob/cdae0b45... which checks the PyPI available file downloads for a package and looks for a bdist_wheel that matches this:
abi_tag = download["filename"].removesuffix(".whl").split("-")[-2] if abi_tag.endswith("t") and abi_tag.startswith("cp31"): has_free_threaded_wheel = True- Note that free threaded compatible doesn't necessarily mean the package supports free threading (concurrent execution), just that it can be loaded into a free threaded interpreter.
This is the case with my own package which is on the hugovk list (apsw) which will cause the GIL to be re-enabled if you load it into a free threaded Python. The reason I provide a binary wheel is so that you don't have to keep separate GIL full and free threaded interpreters around. They have a different ABI so you can't use extensions compiled against one with the other.
Free threading is at the beginning of its journey. There is a *lot* of work to on all C code that works with Python objects, and the current documentation and tools are immature. It is especially the case that anyone doing Python concurrent object mutation can cause corruption and crashes if they try, and that more auditing and locking need to be done in the C code. Even modules in the standard library have only been partially updated.
You can see a lot details and discussion in the comments at https://news.ycombinator.com/item?id=45633311
- Why do you provide it at all then if it's not working as intended yet?
- As I stated:
> so that you don't have to keep separate GIL full and free threaded interpreters around
It means the user doesn't have to keep two Pythons around, install packages in both of them, etc.
It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concurrently modify it in another thread.
- Do you atleast warn users? This sounds like madness.
- Yes. The interpreter warns by default, and requires steps to disable the warning. My release notes say that the GIL will be enabled when the package is loaded.
Is it madness that other packages claim they support running without the GIL, yet it is possible to cause corruption and crashes just by writing concurrent Python code? That is the case with the standard library. Compiler thread sanitizers don't work with free threaded Python. Diligent code inspection by humans is the only way to update C code so far.
Free threading is at the beginning of the project. It works. You get slow downs in single threaded performance due to extra locking, and speedups in concurrent performance due to threading. But it is possible to cause corruption and crashes via Python code. Don't expose it to untrusted data and code.
But do investigate it and see what works well and what doesn't. See what code patterns are now possible. Help improve the tools and documentation. It had to start somewhere, and the current state is somewhere.
- It has never been clear to me what the term "wheels" means in software releases. seems like something along the lines of "thing that works" – does it mean anything more specific than that?
- "Wheel" is the Python-specific terminology for a pre-built package (all non-Python code has been compiled ahead of time, and the necessary object files put in the right places in the package's folder hierarchy), such that installers can simply copy the files (and do a little bit of book-keeping, and possibly pre-compile Python source code to bytecode although the interpreter will do that on demand anyway). The format is originally documented in https://peps.python.org/pep-0427/ .
This contrasts with "sdist", short for "source distribution", which potentially contains code in other languages which must be compiled on the end user's machine (which possibly involves downloading a corresponding build system and setting up a temporary build environment at install time).
Wheel and sdist distributions are successors to a legacy "egg" format. The PEP doesn't explain, but it's commonly held that the name "wheel" is a reference to wheels of cheese, based on PyPI's prior existence as "the cheese shop" (itself a reference to https://www.youtube.com/watch?v=Hz1JWzyvv8A). There might be some ancient discussion on the (now archived and inactive) mailing list to support this.
- It doesn't mean anything useful in this context. It's a leftover from when PyPI was called "the cheeseshop", which in turn was a reference to a Monty Python sketch.
- Cheese wheels?! TIL. Also, for once it isn't a car analogy.
- It's a Python packaging specific term. A "wheel" is a built distribution, which in the most basic sense just means that it can be unarchived directly into a Python import prefix instead of needing a build step (historically `setup.py`) to prepare it for installation.
In practice, many built distributions contain binary artifacts (e.g. builds of CPython extensions). This differentiates them from source distributions, where you'd build the extension from source on your local machine.
- > In practice, many built distributions contain binary artifacts (e.g. builds of CPython extensions). This differentiates them from source distributions
This seems like a good opportunity to mention that that the wheel isn't required to contain any non-Python content, and indeed it's strongly preferred to provide one even (I would say "especially") when there is no meaningful "build step" (https://pradyunsg.me/blog/2022/12/31/wheels-are-faster-pure-...). There are of course abandoned legacy projects around from before the option was available, that may never get fixed. But developers of new code in pure Python should ensure they understand the system.
(I would strongly prefer if we could have a single type of package artifact with an optional post-install hook for building any needed non-Python content. Alas....)
- Yeah, I was simplifying. Wheels are in many ways a strictly better alternative to source distributions even in the “pure” case!
- Wow, long time no see.
psutil is a great project and I do have some future plans involving it.
68 points by grodola 7 days ago | 14 comments