99% of top Python packages are now wheels
pythonwheels.com30 years later I still love programming but people given authority opting for cutesy names for critically important terms in the ecosystem should be severely limited in their options.
Flakes, pills, wheels, eggs... Come on, people. I am willing to bet good money that it won't hurt you physically if you just call them "packages".
I get the issues, but using a fresh name that doesn't have pre-conceptions is often the better idea. If I call something a "package" people will have an immediate understanding of what "package" means, but it will be several slightly different understandings and that can cause quite subtle issues with learning and understanding.
To talk about a different context the BEAM VM (Erlang/Elixir) has "processes". These aren't OS processes but pre-emptively scheduled threads of execution, scheduled by a userspace scheduler onto the available processor cores. They're similar to green threads or fibres in other contexts. They're called "processes" in Erlang because Erlang is old enough that the word process hadn't been implicitly co-opted to mean "OS Process", and it means that whenever I talk about Elixir scheduling I have to insert some version of the above explanation otherwise everyone gets the wrong idea. If they'd called them "florbs" instead, there would be no ambiguity. I still probably have to define the term, but I'm defining it against a universal blank canvas.
Or to put it yet another way "All the good metaphors are already taken"
As a guy who made a career out of Elixir (7 years now, though I have mixed and matched that with quite a bit of Rust and some Golang) I could not agree more on the confusing "process" moniker.
My point was that calling packages "cheeses" and "wheels" was a conscious decision on the part of the Python community and apparently nobody stopped to think if it does not introduce friction or make them look unprofessional.
As you pointed out, Erlang's "process" moniker came from a long long time ago and they have a good reason for it. Python though? 10 years ago many of these problems were well-understood already.
Again, my argument is against cutesy quirky names. I get you that the generic "package" name carries some assumptions with it but IMO that's the more worthy battle to fight: to make sure everyone understands the same thing when "package" is mentioned.
(EDIT: All that being said, Erlang could have indeed used a quirky name because what they have doesn't seem to have its own term yet. And it doesn't exist anywhere else I think.)
> I am willing to bet good money that it won't hurt you physically if you just call them "packages".
So, "Packages are the new standard of distributing packages, and replace packages" is clearer to you?
EDIT: Reduced my original comment to this:
This only outlines that the article is very niche and not very interesting to be posted for non-Python audience.
--- ORIGINAL COMMENT:
Need I explain that if you arrive at such a title you would have to reevaluate if it's worth posting at all? Apparently I need to explain it. ;)
This article is more or less pointless anyhow. Using better terminology would have made that clearer much earlier in the process IMO.
This "article" is a 10+ year-old website that's been tracking an ecosystem-wide migration from one style of preparing python code for distribution to another, more robust style of preparation.
This would have been an awesome introduction paragraph! Would immediately give context to bystanders like myself.
You wouldn't have to say it even
"99% of top Fooz packages are using new package system"
vs
"99% of top Fooz packages are lame"
To consider that the only option is rather reductionist. For example, simply add a version, and to use the original title instead of inventing one:
> 99% of top Python packages are now on version 2
It's not version 2 if it's an independent project.
That's true, but the argument being made was one where "packages" and "packages" was the point of contention. While they're not versions as in numbers, there's plenty of existing work that have similar names, which accomplishes the same goal such as `venv`, `virtualenv`, `python-virtualenv`, `pyenv`, etc. in the Python community.
Yes, but if we just called them packages the headline would be "99% of top Python packages are now packages"
Which would make the article completely redundant, and that works for me. :D
But let's be serious, they could have just said "99% of the top Python packages upgraded to our new packaging system" which would be much more informative and interesting title and would make me read it in full, as opposed to now when I just facepalmed.
> But let's be serious, they could have just said "99% of the top Python packages upgraded to our new packaging system" which would be much more informative and interesting title
How about "99% of the top Python packages don't require being compiled to install them".
That's one of the biggest wins of wheels. They have pre-compiled binaries for all supported platforms of that package. This typically applies to Python packages that have C dependencies.
And I would have never understood that from a cursory look at the article. Learned it just now from you.
So without understanding the reasoning why they are not simply called "python packages" but "wheels", so they have distinct meaning, you chose to go on a tirade about why they are not simply called "packages" instead?
The confidence of some HN commentators never ceases to surprise me
(EDIT: made myself sound less conflicting.)
It's not such a stretch to want to understand things quicker as an outsider and be annoyed by a seeming gatekeeping via quirky witty terminology.
Not sure why you used such a conflicting tone as an opposition of that -- I think, fairly reasonable -- stance.
It's number 3 in "advantages of wheels", which is above the fold on my screen and exactly 50 words into the page, including titles.
Fair enough, my point was that the terminology was a put-off. Reads like kids bragging about stuff and not as a technical article.
But the Python ecosystem reinvents package systems every few years, so that headline is also not very useful.
True, that's why I question the value of those articles in the first place.
That doesn’t make sense either because pure python packages exist and are actively less useful if distributed as wheels
And I have no clue what you just said. :D
It serves only as a further confusion. But yeah, Python is notorious for its numerous (and likely all sub-optimal) approaches to packaging.
The benefit of wheels is that they’re platform specific so that you don’t need to have a lengthy and dependency ridden compile process, you just install the one that works on your OS and architecture and you’re good. Packages written entirely in Python don’t have this problem since the code is pretty much just copied into dist-packages
I see, thank you, that's much better.
I'd still try to shove the platform-specific stuff in a package manager and delegate that complexity to it.
For various reasons that’s often not an option
You're completely right. The site doesn't even bother to explain what "eggs" are. And while it lists the benefits of wheels, there is no word about the apparent downsides of eggs, or maybe a little comparison table.
The official documentation actually has a Glossary so they evidently do know that it's an issue.
We are not even able to align all of this shit in one great platform.
There is 0 reason why all of this is so unaligned.
Alone how many people maintain packages for different Linux distirs etc.
Maintain it once together and only promote it for your distri when you like it.
Soooo much energy wasted
Imagine PackageHub:
Allows for standard features like network of trust, signing, search, smart scanning, transformation into distri specific build flows from a meta format, statistics tracking, CDN, unified package API with dependency resolution and/or SDKs for this.
It would also be much much easier to get sponsoring.
Rpms, apks, debs, msi, exe, tarballs, zips, jars, wars...
Well these are at least more or less known and widely used (though I agree if you never worked with Java that .JAR and .WAR will be confusing).
But when a language ecosystem opts for cutesy niche names then they just end up confusing everybody, including some of their own users.
I think JAR (Java Archive) and WAR (Web Archive) make sense.
But wheel? It's a wheel of cheese, because pypi used to be called cheeseshop? I mean sure, but doesn't it feel a bit like that 20-year-old tatoo that you got during your teens that you got mixed feelings about today?
My suggestion: Call them pypacks -- extension: .pyp
And then everyone will start confusing them with pip.
Though I suppose people already struggle to keep pip vs pip3 vs pip3.9 vs pip3.10 vs pip3.${n} vs pipx vs pipenv straight.
Jokes aside, I feel like the names on python are no more terrible than what Java does.
JAR itself is a cutesy reference. You put your code in a jar, get it? WAR is perhaps not a cute reference, but it hints at what you'll be facing...
That's why I said jar/war are kind of the exception. They are still recognizable for many though, whereas Nix and Python's names are just WTF.
> I mean sure, but doesn't it feel a bit like that 20-year-old tatoo that you got during your teens that you got mixed feelings about today?
That's exactly how I view all these "quirky" names btw.
Ooooo I like that. pypacks
They're not even packages. The package is the source code. A "Wheel" is a compiled distribution.
This is, other than the fact it violates UNIX principles, my biggest issue with Homebrew.
Which UNIX principles do you believe it violates?
Installing files in /usr/local/bin (a default path for all users) by an user that is not root. Absolute no-go.
It hasn’t done that in some time.
For people who are confused by this:
"Wheel has an official standard specification. Egg did not. Wheel is a distribution format, i.e a packaging format. 1 Egg was both a distribution format and a runtime installation format (if left zipped), and was designed to be importable.
Wheel archives do not include .pyc files." [1]
Wtf is an egg?
The other type of package format from Wheel. I agree it's not very clear.
How someone not intimately involved with the ecosystem would figure out all this is beyond me.
There’s probably a blog post somewhere, but Python.org isn’t super helpful when you’re trying to find the equivalent of `cargo run`.
Wheels are more akin to crates, so you'd be looking for cargo add -> pip install.
It's been a while since I've first learned python, but pip install came up fairly quickly. At the time, there were no such things as wheels, but when those came, basically nothing changed for me.
Packages are wheels? lolwut?
> Wheels are the new standard of Python distribution and are intended to replace eggs.
Ahhh... ???
Packages are a language-level concept in Python. A package may be a collection of source files on the filesystem, it's still a package, except that doesn't lend itself well to distribution. Moreover, from the language standpoint, the usage of a package (like "import numpy") is the same on all architectures, but the files needed for distribution (native extension module binaries) may differ.
Python is rife with Monty Python references, such as "spam and eggs" in place of "foo and bar", so "egg" was a natural choice.
And here is the reasoning for naming it wheel: https://github.com/pypa/wheel/blob/3c7b6a0d13ee036ea8f6488ab....
Probably a reference to Monty Python as well (https://en.wikipedia.org/wiki/Cheese_Shop_sketch)
Definitely — the Python Package Index started off as the Python Cheese Shop:
https://tech.slashdot.org/story/05/08/01/0150222/pythons-che...
https://web.archive.org/web/20051027102030/http://cheeseshop...
The linked page has hyperlinks that you can click to get to the specification for at least Wheels.
Pokemon are involved it gets a little confusing.
Is it possible to publish Python code for others to use simply by putting a repo on a webserver?
In other words, can someone put a line like this into their requirements.txt file?
Yes, you can do this: https://pip.pypa.io/en/stable/topics/vcs-support/
That is pretty awesome!
Why is all the "wheels", "eggs", "PyPi" ... stuff needed then?
Wheels are essentially zip files with all the code required to run it contained in one handy little file for all platforms that the package maintainer chooses to build for. Instead of having to download the source of the packages and building on your machines, you have a cache-able neat little file that contains potentially everything you need to use the package. It makes package installations way faster because it skips the build steps and will ship all the binaries required to run the package (ideally). Eggs were this but are not in use anymore in favor of using wheels. Pypi is the package repository for python packages just like npm
Because so much Python code binds to C extensions that you will need to compile for the correct platform. For instance the new version of Pydantic is now backed by Rust. To avoid everyone having to compile the rust code on their platform, you can upload it pre-compiled for linux, windows etc.
Some packages require build-time dependencies not everyone has, like cryptography now requiring Rust. Some packages like SciPy take a very long time to build, and require a Fortran compiler.
pypi is needed because those personal webservers will inevitably go down. why not github? well, it didn’t use to exist before pypi and it microsoft may still pull a good one
eggs and wheels solve binary distribution between other things, so you can install something that requires compilation without having a compiler or the development libraries installed and to avoid bundling files unnecessary during runtime.
a lot of Python packages contain parts written in native code (often C or Rust) for performance reasons.
pip _can_ compile the code at install-time if your PC has the right build tools -- or it can download pre-built binaries for your host in the form of a "wheel".
pypi has a little more metadata than a git repo, and a place to host version-controlled binaries.
New college grad needs something todo?
Probably Ruby owns the award for the worst package names. Can you guess what Sorbet is doing in Ruby ? Or Oniguruma ? Or Ahoy ? Or Figaro ?
Yes, you just need to add `git+` before the URL, see the pip documentation: https://pip.pypa.io/en/stable/topics/vcs-support/
Yes
https://github.com/iree-org/iree-samples/blob/main/requireme...
You can do even better using GitHub releases: the GET for a releases page with assets (ie use inspector) functions as an index url. So you can have a repo that publishes wheels to a release called eg `latest` that can function as the index for that package/wheel. You can even have multiple repos that publish to a single such wheel repo release repo (using the ncipollo/release-action GHA).
I don't know what "GitHub releases" are. And I don't want to know :)
I prefer to use as few technologies as possible. And completely stay away from propriatary technologies.
> I prefer to use as few technologies as possible. And completely stay away from propriatary technologies.
This is the most boring and tedious archetype out there.
you can install a package straight from Github:
or from the git repo:pip install https://github.com/whalesalad/verisign-namestudio/archive/refs/heads/master.zippip install git+https://github.com/whalesalad/verisign-namestudio.git
What does this mean?
The three that are not wheels are distributed as source tarballs only -- one is AWS Sagemaker that looks like it uses a bunch of native code, one is Apache Spark, and one is futures.
I think the news here is that there are no packages in the top N that are using an alternate packaging system without using wheel as well. (alternates being msi, exe, or egg)
"new standard" apparently is a PEP from 2012
Yet most projects dropping 32 bit wheels.