For whatever reason (maybe post 2to3 PTSD), Python community seems not extremely eager to jump on latest versions of Python and it often takes a long time for popular libraries to support the latest and greatest, so I’d recommend patience and baby steps
Well, you'd think after the 2 to 3 debacle, python might take backwards compatibility more seriously, but they don't.
Follow semver, and stop breaking things on 3.x. If it's deprecated in 3.x, don't remove it until 4.
PEP387 requires that introduced incompatibilities have a “large benefit to breakage ratio,” and that any deprecations last a minimum of two years.
FWIW, Kubernetes has a similar approach. Breaking changes occur all the time with “minor” version updates.
E.g. Python 3.12 has deprecated datetime.utcnow(). So it will probably be removed in Python 3.14 or 3.15.
For all intents and purposes, "Python 3" is now the brand and "Python 3.x" releases are major, breaking releases.
Clojure is pretty good even on that timescale though.
[1] See eg https://stackoverflow.com/a/50445603 up until 2021
In some environments, yes. A bunch of platforms have started using bash-compatible (but not actually bash) shells like fish to help with startup performance. Apple has upgraded from a truly ancient version of bash to a somewhat-modern one in that time (and then gone all-in on zsh instead). Things change on the scale of a decade.
Yes, of course, things do change incompatibly in a decade, probably including some of the nonstandard platform-specific commands called by the script regardless of which dialect of shell. But people who use a nonstandard shell dialect - however common it might be at the time of writing - should be prepared for that shell to become uncommon, absent by default, installed at a different path, incompatibly changed, or otherwise unusable a decade later to an extent that people writing to the standard do not.
It's harder to upgrade services in a central way with any amount of leverage, and generally requires more coordination overhead, and moving more carefully.
Compare with, say, golang, where it's pretty much a non-issue. My experience with Ruby was a lot better too, until Ruby 3, but hey, that was a major version bump!
People who have Python 3 installed can be on many different versions. The thing is, depending on the version, quite often bug fixes included in later versions aren't in older versions. So if you want to make your code work -- got to get the patches in manually, monkey patch broken code, and do it that way. Then there's the seemingly random deprecation of standard library modules / other breaking changes.
I take python version support seriously because if people install your packages you'll be outsourcing all of the above crap to the user. They might not even know how to 'upgrade' python. Or end up on the wrong version. If your package doesn't work when they install it they'll just move on to something else. Python is a total shit show for packaging.
Doing development on the projects without a specialised build environment, though, is almost hopeless. You either have to manually install a bunch of stuff (e.g. I use ruff, used to use ruff and black, used pre-commit, and a few other things), and then there's the non-Python things that need to be installed (like specific fonts in my case). At this point I just impose Poetry as basically a requirement for development.
The program does build and function in a non-Poetry environment which is what I use for production.
Moving fast is really nice when you can get paid for shipping and have the project either be retired or someone else’s problem in a decade.
Most places don't have the resources to keep updating everything for little benefit. (just look at node its terrible to support. )
I don't think it's the 2 -> 3 thing any more, that was a while ago. Honestly there are just a lot of things that don't work well in 3.x.0 Python releases. For example, 3.12.0 had the per-interpreter GIL thing; I tried that in 3.12.0 and ran into a completely breaking issue almost immediately. They were responsive & helpful and did fix the first issue in 3.12.1, but we still had more issues with parts of the C API which seemed to work in 3.11 and better again in 3.13, but it really felt like that needed another release to solidify. (Also you can't import datetime in that setup in 3.12, which is also a pretty big deal-breaker).
I can only imagine the free-threading thing will need at least the same kind of time to work the kinks out, although it is nice to see them moving in that direction.
I suspect this is a mix of Python 3 being “good enough” for most cases and companies not updating their stacks that often. I think most of us came into Python professionally around 2.7 so the need to keep updating our version hasn’t been heavily ingrained into our thinking.
Single threaded cpu bound workloads suffer in benchmarks (vs i/o workloads) till they put back the specializing adaptive interpreter (PEP 659) in 3.14. Docs say a 40% hit now, target is 10% at next release.
C extensions will have to be re-built and ported to support free threaded mode.
Some interesting and impactful bits of open source work for those with a c++ multithreading background.
So far in production if I need to use multiple cores, I use multiple processes and design apps that way. The discipline this imposes does seem to result in better apps than I wrote in an environment like Java with tons of threads.
Huh? I python you have to choose either threads or asynchronous/await? Why not combine both of them? I am so confused. C# allows for both to be combined quite naturally. And JavaScript as well allows for workers with async/await.
Seems to suggest that threads and yield/async were mutually exclusive. I misunderstood. I will move on.
Weakness may have turned into a strength.
There are all kinds of micro-optimizations, as in: one has to know which Pandas operations are going to be more expensive than others, and organize the code accordingly, but these things often teach programmers the wrong ideas. It's not uncommon in Python world that a superior solution (from algorithmic perspective, i.e. the one that should use less time or space) is in practice inferior to a solution that's implemented in C. And so, writing more efficient Python code comes down to knowing which functions are faster or cheaper in some other way, but it doesn't generalize and doesn't transfer to other languages.
What usually happens in situation like this is that the developers of the language (or a product, a framework etc. that suffers a similar fate) start optimizing the bad solutions (because they are the go-to tool for their users) instead of actually improving the language (the product, the framework etc.) To give some examples of this happening in Python: there's a lot of work dedicated to the performance of lists and dicts. But, if anyone really wanted performance, they'd have to look for more specialized collections, rather than optimizing very generic ones.
If you take the time to split up your data into chunks you can avoid the GIL entirely with multiprocessing. Or by handing it off to a library that does it for you. just as long as they’re not using the same python objects.
For example, if I parse an XML document with ElementTree, as a quick experiment, parsing the document takes ~1 second and serializing all the elements names and attributes to JSON takes an additional ~0.5 seconds. Serializing the whole ElementTree object using pickle takes ~4 seconds. Serializing it as XML takes roughly as long as parsing it.
It also screws up C extensions that don't support fork and does not solve the problem of increasing TLS handshake/fanout in outbound requests by the number of cores.
The whole thing sucks in production.
Is threading potentially better for IO bound tasks than async?
Yes, but Python now has a version without GIL, which prompted this post in the first place. So my question is: Now, if I use a version of Python 3.13 without GIL, can a threaded Flask app do better than an AIOHTTP server.
So for your original question
> Is threading potentially better for IO bound tasks than async?
async will be better in general, potentially due to Async co-routines using far less memory than threads and being better under high concurrency. But that's all will depend on the details of implementation of the server.
Its a pick your poison type of deal.
I _personally_ hate async. Its a suprise goto, which I don't like. However in my testing asyncio performs the same as threading. (this was with socket based IO)
If you loose the GIL, then threading _should_ be much faster, as it should scale to how many cores you have. But thats only if you have enough IO to saturate a core.
Asyncio can run tens of thousands of tasks if its used properly. If you think something will block it you should check out "process pool executors." Note that its very tricky to share resources like sockets between processes so its kind of another reason to avoid stuff like this.
I think Python 3.14 will have interpreter pools for even more concurrency options.
> If you think something will block it you should check out "process pool executors." Note that its very tricky to share resources like sockets between processes so its kind of another reason to avoid stuff like this.
Isn't that the benefit of no-gil? The ability to run CPU-intensive operations without incurring the overhead and friction of multiprocessing? Now you can do multicore processing while also having shared memory
I found that (i) was blocking (ii) making the image sorter unusable.
So far though that is processes and not threads.
For the last few weeks for the hell of it I've been writing a small and very pedagogical chess playing program in Python (trying to outdo Lisp) and once I got the signs figured out in the alpha-beta negamax algorithm it can now beat my tester most of the time. (When I had the signs wrong it managed to find the fool's mate which is not too surprising in retrospect since it looks ahead enough plies)
That was my major goal but I'd also like to try an MCTS chess program which is more of a leap into the unknown. Unlike alpha-beta MCTS can be almost trivially parallelized (run 16 threads of it for, say, 0.1 s, merge the trees, repeat, ...) and threads would be a convenient way to handle concurrency here although multiprocessing out to be good enough. So I am thinking about using a non-GIL Python but on the other hand I could also rewrite in Java and get a 50x or so speedup and great thread support.
(Note the problem here is that unlike games where you fill up a board, chess doesn't really progress when you play out random moves. With random moves for instance you can't reproduce White's advantage at the beginning of the game and if your evaluation function can't see that you are doing really bad. I need a weak player for the playouts that plays well enough that it can take advantage of situations that real players can take advantage of at least some of the time. A really good move ordering function for an alpha-beta search might do the trick.
Has anyone started deploying nogil at scale in prod?