Should Rye Exist? · astral-sh rye · Discussion #6

Are there any synergy efforts between the development of the fantastic PDM and that of the brilliant Rye?

The docs of rye mention it is using pip-tools for locking. A major difference between pip-tools and Poetry is that, poetry creates a platform independent lockfile.

pip-tools creates a lockfile for the exact os and cpu architecture you have run it on.

Python packages can specify a different dependency depending on cpu architecture . See this tensorflow isssue for example.
Tensorflow brings totally different packages for any amd64, MacOS arm64 or Linux arm64.

If you create a lockfile containing tensorflow with pip-tools on Apple Silicon , you won't be able to install it on linux

Afair, resolving something platform independent with wheels is not really 100% supported via pypa standards and poetry applies bunch of educated guesses where to find this information. Its one of the reasons for it being slow.

A major difference between pip-tools and Poetry is that, poetry creates a platform independent lockfile.

That is rather off-topic for this discussion about whether rye should exists. Maybe open an issue, but afik, pip-tools is just the stand-in option and another dependency resolver and locking tool will be used in a stable release.

I was addressing the above question in the sub thread. But if you mention it, I would argue, rye absolutely should exist, but only if it can hold up to solutions where such issues are solved. Cargo is mentioned a lot on this thread, but also cross platform lockfiles are a solved thing in bundler for a decade.
https://nokogiri.org/tutorials/installing_nokogiri.html#supported-platforms

Well, that's Rust and Ruby. Python packaging has a few complications, so it's not an entirely solved thing, because distributing interpreter versions is not a solved thing and the whole packaging system is a perpetual work-in-progress. Again, pip-tools is not the final solution, and just used because it was the quickest at the time. A cross-platform lock-file is, to my knowledge, planned, but difficult to do right in Python. Poetry imho is not a good candidate for a locking engine in rye because it is very opinionated, to heavy-weight and doesn't play nice with others.

There are PEPs trying to address some of the problems listed above:

• A proper lockfile standard

PEP 665. Rejected due to the issue of sdist.

• A way to fetch metadata from indexes, without having to fetch wheels

PEP 658. Accepted but the implementation progress is slow.

• Unzippable binary distributions of all major Python interpreters on all major platforms

PEP 711. Draft

@mitsuhiko Could you elaborate the sentence above, trying to understand what you mean by that:

py I think is also a problematic entity, because it's tied to cpython's distribution.

IIUC rye is tied to the python-build-standalone distribution, is there any condition that the official Python distribution on python.org would need to make py not problematic anymore?

py I think is also a problematic entity, because it's tried to cpython's distribution.

That depends on which py you're referring to. The Windows one comes with CPython, the Unix one that I created does not.

I'm also not sure if py actually plays a role today in the wider ecosystem. I at least haven't ever used it, and I don't remember it coming up in documentation of packages I use.

You're right that it doesn't play a wider role. I have some long gestating ideas on letting people participate in the interpreter search and support for subcommands, but I haven't had the time yet to make them happen. Only at that point would py (for Unix at least) start to potentially participate as something other than a launcher of Python interpreters.

The problem with that system is that there isn't a clear set of defined boundaries between those tools which allows one to work with another. There isn't also an overriding interface which allows beginners to start safely using whole infrastructure in a simplified way and then later transfer to using the particular tool for an area if they want some specialist feature.

A good summary is How to improve Python packaging, or why fourteen tools are at least twelve too many - note the author of that is part of this discussion above. Not so long ago did a "try again" with pipenv, assuming that since it was now being recommended as a standard tool it should be working. This turned out not to be the case at all and had to be backed out. That just shouldn't happen with the "official" tools.

pipenv, assuming that since it was now being recommended as a standard tool

FYI pipenv isn't "official" beyond the fact the maintainers have it under the pypa org on GitHub which isn't more than a group of projects sharing some infrastructure. Hatch itself is also under the same org, for instance, so pipenv isn't the official workflow tool for anyone except projects that choose to use it. The only requirement for a project to join the org today is for someone to ask if a project can join and other pypa org project maintainers vote on the suggestion (pipenv predates this policy BTW).

I did get that clear, but some time after having read that it was the official tool and deciding to try it. It was unable to work for my needs even with considerable willingness to find clever workarounds.

In fact, I should say specifically, that it was the above essay about fourteen tools being twelve too many which allowed me to realise that despite claims otherwise pipenv was in no way "the" official (as the Zen of Python) way of doing things.

Pretty sure pipenv suffered a lot from having "a maintainer of all time". It wasn't the official tool and didn't have community consensus that it should be, but was advertised as the official tool by a private overreach of authority.

However that's in the past as it is no longer advertised as such?

mostly because the PyBI builds try to retain this framework structure for macOS that's just not needed

It's definitely needed for a whole slew of use-cases; basically anything that calls a platform API on macOS.

@glyph I would actually be curious for when a framework build is needed. As @indygreg mentioned a lot of macOS APIs work fine in a non framework context.

As @indygreg mentioned a lot of macOS APIs work fine in a non framework context.

Some do, some don't. Very few (beyond UNIX-y and CoreFoundation ones) are documented to do so; there's just a strong assumption throughout the platform frameworks that you exist in a bundle (usually an app), and that your frameworks do as well. So it's very annoying to research specific APIs because Apple won't tell you. You have to build your own examples, and then you get arbitrary undocumented failures about Mach ports which boil down to "did you build as a bundle in an app / framework? no? oh, that's your problem".

But let's assume that all APIs are actually callable from a non-framework build. If you want to distribute software for macOS, you can't do so, because all the code-signing tools break. If you want to distribute the build of Python itself, for example, unless you live entirely in CLI-land and opt out of quarantining (i.e. "files you downloaded from a web browser") entirely, it is not possible to distribute a stand-alone executable that lives outside of a bundle. Individual binary files can be code-signed, but they cannot be notarized, because there's nowhere to attach the notarization ticket. In other words, you have to end-run the entire security apparatus of the platform, and rather than it being a temporary issue for lack of attention, standardizing on non-framework builds takes a step further towards cementing it as the permanent status of Python.

If you try to distribute an app bundle using a non-framework build of Python, then you suddenly run into a bunch of other issues related to the way that codesign tool respects bundle resources and deals with dynamic linking. None of these are totally insurmountable — it is possible to ship a raw dylib in a framework bundle (and in fact Python binary wheels do have this problem as well), but it breaks a bunch of assumptions from tools like py2app that want to link against Python as a framework, and codesign, stapler, and notarytool.

Then there's the issue of callbacks. If you want to include Python as a dependency of a native Xcode project, there's no way for Swift or Objective C tooling to consume "arbitrary directory with a pile of UNIX-ish junk in it"; they expect to see a Framework with Headers, Versions, et. al.

If you ignore all of this, then sure, you can limp along and kinda call a bunch of APIs and kinda muddle through to make something distributable, but you cut off all kinds of possibilities and make natively accessing the platform arbitrarily weird and difficult and broken in edge-cases for no particular reason; we already solved framework builds and they work fine. Getting rid of them would reinforce the idea that Python is only for backend/web stuff because it's just too confusing and broken on client platforms, and the Mac port would regress further towards treating macOS as a slightly-broken version of Linux rather than its own platform.

You have to build your own examples, and then you get arbitrary undocumented failures about Mach ports which boil down to "did you build as a bundle in an app / framework? no? oh, that's your problem".

Case in point: CoreLocation used to be my go-to example of an API that didn't work outside of a bundle context, but it looks like maybe that does work OK now? NSUserNotificationCenter also works now but didn't used to, but it's deprecated now and in a quick test (which could be broken, btw, it's annoying to switch back and forth between framework and non-framework builds and set up the appropriate context) its un-deprecated replacement UNUserNotificationCenter appears to silently do nothing. I can't quickly find a "crash with some message about mach ports" example but there are like a hundred frameworks on the platform and I only have examples for a few of them, so this is unfortunately more like folk wisdom.

Oops, I double checked, and looks like I did in fact use a framework build by accident. CoreLocation no longer spits out an error message on a non-framework build, it just segfaults immediately.

What don't you like about lockfiles?

The time taken to lock + locking os-dependent files.

Lock files are optional and very much necessary in enterprise (production) contexts. No one is a big fan of how much time it costs to resolve an entire transitive dependency tree, and for most hobbyists it's overkill, but in the interest of software reliability, you really can't use python without them (until ideas like that of nix take over all of computing, which likely will never happen).

Why is the comparison here to rustup, cargo, etc.? Why aren't y'all comparing to ruby which is the same type of language but with better tooling? gem and bundler have been standardized in Ruby for decades now and have no competitors. gem is vastly easier to use than pip. Everyone using ruby knows how to use these tools, and they work perfectly. Cargo is literally written by the creators of Bundler. Use Bundler as your base level of what a tool should do in a dynamic language and then maybe move on to Cargo from there.

Cargo is literally written by the creators of Bundler.

Thanks, now I finally know the underlying reason why I've hated cargo for so long.

Thanks, now I finally know the underlying reason why I've hated cargo for so long.

And yet here you are in a thread for a language that has spent 30 years and still doesn't have working package management or install tools. The tool that works and is standard beats out the tool that doesn't exist.

But... we literally do have working package management and install tools? And they are flexible enough to let people choose how they want to install things, without preventing opinionated tools from choosing a single way and sticking with that everywhere.

No one is worried about whether python has working package management or install tools. The hot topic of the day is whether python can have a "one tool to rule them all".

you really don't. The tools are all garbage and hardly work at all across systems. I used python for years at my last job (and still partially use it at my current job), alongside ruby, java, kotlin, and rust. The difference is stark between the two. Claiming that other tools are 'opinionated' and that python has choice is just laughable. I'm sorry, but Bundler isn't opinionated, cargo isn't opinionated, maven isn't opinionated, gradle isn't opinionated. They're much more powerful tools that avoid the problems that literally every python tool deals with. You act like these other tools have limitations just so you can justify having 'choice' in the python community. I'm sorry, but it's just not accurate.

The tooling in the python world has always been terrible. Acting like you have a choice when the community recreates the package management and build tooling every year is just disingenuous. It just indicates that none of the tooling actually works. There's always a shortcoming.

The first step is admitting there's a problem.

Me neithers, it is somewhat hard to say, who the bulk of Python developers are and how they came to Python. There's the concept of "dark matter developers" and coming partly from industry and partly from academia, I see a giant lot of those. You came to Python rather early and are, even without well-deserved flattery, somewhat exceptional. As is the whole "early wave" of Python users including devs. Just by virtue of Python's rapid growth in recent years, nothing early can be terribly representative of the community, now. But yeah, I'm not completely against C or Rust, all I'm saying is, if there's the choice between Python native and one of those other languages, tooling should lean towards native, unless there's a very good and compelling reason, because people who know Python will have a lower barrier of entry and that has all sorts of advantages when we speak of Python tooling.

That is no critique of your choice of language for rye. It is your project and you are free to judge, what the best tool for you to do the job is. Rye is very very interesting to me and elegantly solves a bunch of problems already, that I care deeply about. Rye clearly should exist for that reason alone. I'm just saying, that I hope whatever tool becomes the ultimate packaging and distribution tool, will be not need to be written entirely in something other than Python. I also see how rye could inspire that ultimate "winner", and that's why I'm here rambling in the first place. Since is done with experiments, and experiments are often repeated. That is how we learn, improve and come to robust knowledge.

unless there's a very good and compelling reason

Speed. Linters, dev servers, and similar tools should be fast enough that the wait time is barely perceptible.

Speed. Linters, dev servers, and similar tools should be fast enough that the wait time is barely perceptible.

Sure, speed can become such a compelling reason. There's studies about that kind of thing: everything under half a second is absolutely fine, 1.5 is where it begins to be mildly irritating - you'll be loosing people to internet and coffee when it gets to the minutes. As I said, I'd argue everything other than Python should have a compelling reason and most of the time speed is overrated as one. In addition, some tools simply can't be made much faster. Pytest can't test Python faster than the interpreter can run the code. Network code, can't get data faster, then the communication allows and even than understanding and changing your tools without learning an extra language maybe worth more to people than even a 50% increase in speed in some situations. In any case, we're going offtopic a bit, arguments stand, lets move on, what say you?

The Rust in the Python ecosystem now displaces Cython, C and C++, not Python code.

I think that's true overall, but using Python to prototype something which you then later choose to re-implement in Rust is also a totally legitimate use case for Python as well. It's quite possible the prototype works well enough that you can leave it in Python, but it's also possible you simply need to squeeze every CPU cycle that you can get (although watch out for premature optimizations). But both scenarios are totally fine and use Python for its legitimate strengths of being productive to code in.

@con-f-use I totally agree

1.5 is where it begins to be mildly irritating

When the VS Code Python extension had a bug that delayed file saving by 1.5 seconds, I was more than mildly irritated, after like half an hour of working like this, I needed a break to not get cranky.

And I don't buy your argument as a valid criticism, tbh.

Tools should be written however the tool author considers productive. That may be an indictment on the python language if the author doesn't consider python to be productive. That may raise the barrier to entry for other users to contribute to or hack on the tool. Both of these factors are not really pertinent to the question "oh, tool author, why did you make this bad mistake when choosing an implementation language".

But also, my original point was exceedingly targeted. pyenv is software for installing python executable interpreters, and the authors of pyenv felt that the best language to write a tool that installs python for you, is... NOT python.

There is a reason for that, and it's perhaps the most important reason of all: you can't install python with a program written in python, if you do not yet have python available. A program written in a compiled language that builds into a single self-contained executable file is well-positioned to handle acquiring python.

It is indeed silly to suggest writing pyenv in python, and likewise I consider it silly to suggest writing rye in python... unless rye first deletes all functionality related to installing and managing python installations.

Another valid approach might be as suggested elsewhere in this discussion: divide rye into two tools, one to manage python installations written in rust, and one to run inside a python installation and manage python packages, written in python. You likely only care about hacking on the latter, anyway.

So you can't compile gcc with gcc, huh? Interesting... 😉

It is the nature of binary distributions, that you can distribute compiled binaries, i.e. the compiling of the python runtime does not have to happen on the endusers machine. It could be bundled with the distribution binary of rye.

Sophisim aside, I was talking in general, and you are making a very rude and narrow point. Python tooling, if no compelling reason not to exist, should in general be written in Python. Exceptions exist. That is my opinion, you are allowed to yours. My criticism is as sound and valid as yours. I maintain, the original question was not particularly silly, and in any case calling questions silly isn't good form.

To re-iterate: my argument in favor of writing Python tools is Python is an accessibility and ownership one. I want the community to contribute, I want them to be represented by the tooling, I want them to participate, I want to reduce the barrier of entry. Having to learn another language, having to duplicate and re-implement and use tools specific to that other language hinders that. What makes my arguments that trying to write Python tooling in Python helps with that (in most cases) invalid or unsound?

So you can't compile gcc with gcc, huh? Interesting... 😉

That's a fascinating topic, and there are people who say the answer is "no": http://bootstrappable.org/projects.html for details on how they compile GCC in a multi-stage pass that begins with a stripped down C compiler capable of building a "more fully fledged C compiler like tinyCC".

It is the nature of binary distributions, that you can distribute compiled binaries, i.e. the compiling of the python runtime does not have to happen on the endusers machine. It could be bundled with the distribution binary of rye.

You could, and doing so would be quite unwieldy. Shipping a python just so you can run a tool that installs python? Single-file executable handling is not very fast either. And you remove much of the benefit of writing it in python, because actually building the binary you hacked on is a lot trickier than it is in an ecosystem where single-file executables have native support.

What makes my arguments that trying to write Python tooling in Python helps with that (in most cases) invalid or unsound?

The fact that my comparison, pyenv, made it pretty clear that my opinion was and still is, that distributing a python program which is used for installing python, is not very ergonomic -- and criticizing a project that avoids python for this task is... pretty silly. The project has a compelling reason, the compelling reason was part of the original problem statement, so I don't see where

Python tooling, if no compelling reason not to exist, should in general be written in Python.

comes in. Disregarding the compelling reason and then trying to make this a generalization for virtualenv managers (a number of which already exist, written in python no less), doesn't make my original criticism of a specificity suddenly turn into a criticism of a generalization.

It makes a lot of sense for tools such as pip, pdm, poetry, hatch, etc. to be written in python, and I absolutely agree with you there. Fortunately, they already are written in python.

It makes a certain amount of sense for rye to focus on acquiring python using rust code, and dispatching to python code once it has a python available. This would empower users to hack on the project management routines as long as they don't need to hack on the "acquire python itself" step, using nothing other than knowledge of python itself.

Getting rid of all rust code seems to be very much missing the original point of this project and it is disappointing to me to see people seriously suggest it.

Now that is a line of reasoning I can live with. To be honest, I antagonized you because the "silly question" think and general superior air rubbed me the wrong way. But just because I enjoy discussion (which is euphemistic for "I like to argue"), I'd still say, it's kind of a valid point still. It's not that much harder to bundle a standalone python interpreter with rye, than it probably would be for future prospect contributors to learn a whole other language, just to be able to contribute. Rust was probably the more practical and easier choice for Armin and that is why he made it. But the whole "should rye exist?" question is predicated on the community as well and I'd prefer the ultimate packaging tool to be governable and accessible by the whole community as much as possible.

That's a fascinating topic, and there are people who say the answer is "no"

Right? I think so too and have spend hours of fun research. It is, however, a rather academic ivory-tower subject. The fact remains, that I can download a gcc binary that runs on my machine, I can download the code of gcc and then after a long unnecessary tedium of yak shaving, I'll end up compiling the gcc code with my gcc binary on my machine. Same goes for bundling a standalone version of the python interpreter, that work has been done and Armin is using it from another project to install Python interpreters. If you ship someone else's pre-compiled interpreters anyway, you could feasibly ship them with your interpreter install tool, too.
Mostly, bootstrapping problems are a little like watching those preppers who have a lot of fun and make entertaining videos imaging they'll have to re-build civilization after the apocalypse has come and we have regressed into the stone age. How on earth would we do it? What would we have know? With how little can we start? 😁
Security vetting, novel platforms and traceability concerns aside of course, which is were solving the bootstrapping problem is of real-world import. Anyhow, thanks for not escalating my rather aggressive line of argument, I hope you enjoyed yourself as much as I. Let's not go further offtopic here. I feel mildly bad for hijacking Armin's question. Luckily github doesn't charge by character (typographic or human).

But just because I enjoy discussion (which is euphemistic for "I like to argue"), I'd still say, it's kind of a valid point still. It's not that much harder to bundle a standalone python interpreter with rye, than [...]

Eh... I hear the argument, I disagree (not too fond of e.g. extracting a hefty python payload to /tmp at every startup), but I acknowledge that opinions can vary here.

I hope you enjoyed yourself as much as I.

It is, indeed, a fun topic. Especially the novel platforms bit.

@csaroff if the shim is installed then python automatically picks up the right virtualenv. The same is true for rye run script which also automatically enables the virtualenv for that script. What I don't want is to create a shim for all potential scripts and mess up the global path.

@marcelarie

True, npm doesn't cleanly handle projects with different node versions which can be annoying.

It does handle your package deps(node_modules) though!

with asdf you don't have to worry about any of this. cd to the directory, all your tools activate for every language.

Tools like asdf don't necessarily solve usage from scripts. The relationship between the working directory and your packages is natively understood by node/npm (because they're looking for node_modules in or around CWD), but Python requires the use of a specific interpreter. Can you run asdf in a script? Does it always work the way you expect, even in things like CI/CD pipelines? Those aren't questions you need to worry about often in ecosystems that didn't go the venv route (i.e. most of them).

Can you run asdf in a script?

you don't 'run' asdf. you just let it handle setting the environment when you're in a specific directory. And yes you can easily refer to the active shim by just referring to the shim in the ~/.asdf/shims directory, but you don't really ever need to do so. asdf's power is that it just works. If I have a script that refers to python and that script is run from a repo that has a .tool-versions file, it will use the version from that file. For example every single one of our repos has a Makefile that calls out to a shell script. These shell scripts can refer to any tool and if that tool is declared in the .tool-versions file then it will just work. There's no setup, no version management, nothing. It works both locally and in CI/CD.

In regards to usage in CI/CD, yes, it's quite simple, even with caching. https://github.com/asdf-vm/actions

@pauleveritt my main opinion is that it has no traction, so there is no point going down that path right now. There are more important things to figure out.

for that you can use pyenv or asdf, for example.

I made a tool for handling that piece specifically @th5: encant