Why did Nix adopt Flakes?
jetpack.ioMy biggest issue with Nix flakes right now is the way it integrates with Git.
It insists on copying the entire repository into the Nix store (which makes all of its content world-readable!). Even if your flake.nix is in a subdirectory of a monorepo, the entire monorepo will be copied into the store every time!
It's a known issue but fortunately is being worked on: https://github.com/NixOS/nix/pull/6530
Doesn't this still copy the repo, but just doesn't unzip it?
I think it only lazily copies files that are referenced by your nix expression.
I tested this branch, and a flake-based shell launched in 3 seconds inside a large monorepo (vs 30 seconds before). I don't think it is possible to copy this repo in 3 seconds.
I hope that helps with nixpkgs repo as this takes a lot of time whenever it is updated.
I don't think it will. There isn't any good way to incrementally fetch remote nix files, so to use nixpkgs it all has to be downloaded and unpacked.
Unless you mean when developing on the nixpkgs repo, in which case it could help.
ah, yeah, we have a cheeky workaround for this in Devbox ;-)
https://github.com/jetpack-io/devbox/blob/main/internal/impl...
Yeah this is annoying purely from a disk space perspective too as I easily go through 200GB of disk space in a day if I'm actively working and iterating on some Nix code
Isn't it the same with channels though? On my system `/nix/var/nix/profiles/per-user/root/channels/nixos` is a symlink to `/nix/store/[...]-nixos-23.05/nixos`.
Yes, but channels are updated less often and it's very unusual to put an application git repository into one.
You might have two dozen flakes in your system. You'll typically only have a single channel, if you use them.
Is this still an issue when using gitignoreSrc?
For me, yes. Even with a completely clean checkout, the repo is a few hundred megs.
I'm a casual nixpkgs contributor. Flakes are like the embassies Nix sends out into the OSS world. Discussion about building the application are kept with the application, nuances and patches can be discussed with the actual authors.
Remember the debian SSH packaging snafu? The application authors weren't involved. (edit: see below)
Nixpkgs is like the state department, a central unifying hub, great to bootstrap the package ecosystem (ten years old now), but it needs to spread its wings.
>Flakes are like the embassies Nix sends out into the OSS world.
It's a package spec, just like debian/* and rpm/*.spec.
>Discussion about building the application are kept with the application, nuances and patches can be discussed with the actual authors.
Every distro package ecosystem has the maintainers discuss patches with upstream.
>Remember the debian SSH packaging snafu? The application authors weren't involved.
Assuming you're referring to the 2006-2008 OpenSSL vulnerability (which affected openssh and other things), yes the openssl devs were involved. The Debian maintainer who added the bad patch asked about it on the openssl mailing list, and only added it after he got ACKs from the devs.
> It's a package spec, just like debian/* and rpm/*.spec.
No. It's not. The interface that Nixpkgs' callPackage expects to find in default.nix within the Nixpkgs source tree is (kinda) like an RPM spec or Debian control file and friends, but that's not what flakes are.
Flakes is mostly about providing mechanisms for distributing and referring to code written in the Nix language, and in that way facilitates the maintenance of 'out-of-tree' (read: not in Nixpkgs) Nix packages. But flakes can also be (and are) used to ship Nixlang libraries, end user configurations, IaC for clusters or fleets of servers, overlays of packages to be composed with an end user's copy of Nixpkgs, etc.
A flake doesn't have to contain any packages at all. And when it does contain a package, the package is always defined with reference to some other package collection that it pulls its dependencies from (almost always Nixpkgs), and the conventions for describing the build come from there. They're not part of the flakes schema.
> Every distro package ecosystem has the maintainers discuss patches with upstream.
Again, not what flakes are about. The situation the GP is describing is where a Nix package's 'recipe' lives inside the upstream repository* and is maintained there directly by contributors to/maintainers of the upstream project. This is more like a project including a Dockerfile for the convenience of new contributors, or application developers directly publishing their own Flatpak images.
>Flakes [...] facilitates the maintenance of 'out-of-tree' (read: not in Nixpkgs) Nix packages.
Yes, aka a Nix package spec.
>But flakes can also be (and are) used to ship Nixlang libraries, end user configurations, IaC for clusters or fleets of servers, overlays of packages to be composed with an end user's copy of Nixpkgs, etc.
Yes, just like DEB / RPM packages.
>A flake doesn't have to contain any packages at all. And when it does contain a package, the package is always defined with reference to some other package collection that it pulls its dependencies from (almost always Nixpkgs)
Yes, what DEB / RPM call meta-packages.
>The situation the GP is describing is where a Nix package's 'recipe' lives inside the upstream repository
Yeah, and just like you convinced upstream to hold your Nix package spec, many upstreams hold Debian / RPM package specs too.
It's a defined schema for exporting nix functions I'd say. I could share a flake with you to provision your disks like I do, a service like I do, your cloud infra or anything else nix can do now. It is way more than deb, but if you only compare packaging it'd be like automatically building every PPA and every locally compiled pkg against exactly your installed version of every pkg it uses.
> > Flakes [...] facilitates the maintenance of 'out-of-tree' (read: not in Nixpkgs) Nix packages.
> Yes, aka a Nix package spec.
Right... one of things that flakes can be used to distribute is roughly comparable to an RPM spec. That is not the same thing as being an RPM spec. A BitTorrent magnet link used to distribute an MP3 file is not itself an MP3 file, either.
> > But flakes can also be (and are) used to ship Nixlang libraries, end user configurations, IaC for clusters or fleets of servers, overlays of packages to be composed with an end user's copy of Nixpkgs, etc.
> Yes, just like DEB / RPM packages.
There is a sense in which flakes and RPMs can (sort of) be used to achieve the same things here, but not remotely in a way 'just like' each other. A flake.nix file is not an archive, and an RPM is not a configuration file format for something like Puppet. The latter point is so obvious that I have to assume you're being deliberately obtuse here.
> > A flake doesn't have to contain any packages at all. And when it does contain a package, the package is always defined with reference to some other package collection that it pulls its dependencies from (almost always Nixpkgs)
> Yes, what DEB / RPM call meta-packages.
No. RPM and DEB metapackages are 'abuses' of an archive format for either (a) just distributing install-time hooks and activating triggers or (b) forcing the installation of actual packages through the normal mechanisms of declaring dependencies on them.
Nix has no equivalent to (a) because 'installation' doesn't mean the same thing with Nix and hooks and triggers are owned by Nix profile managers rather than packages.
If Nix has anything equivalent to (b) in the RPM and DEB worlds, it's this¹ which has nothing to do with flakes and is certainly not equivalent to a flake with no packages declared. In fact, any flake used to distribute such a buildEnv-based metapackage would have to declare it as a package.
A flake without any packages declared is not an archive of a Nix package whose data is empty but whose metadata is present.
Idk what to tell you. Clearly you are trying to somehow 'demystify' Nix terms by equating their referents to things that are familiar to you and others, but your desire for succinctness here pushes you to elide basic differences— not just with respect to Nix but with respect to source-based package management in general. Just like with two natural languages, it's simply not the case that every term here has a 1:1 translation.
--
1: https://nixos.wiki/wiki/FAQ#How_can_I_manage_software_with_n...
Yes the OpenSSL one
https://lists.debian.org/debian-security-announce/2008/msg00...
> he got ACKs from the devs
You're totally right: https://news.ycombinator.com/item?id=6343782
> Every distro package ecosystem has the maintainers discuss patches with upstream.
I don't think that's entirely fair to say, it's certainly best practice.
I love the analogy!
I think this is a great strength of flakes. Packaging an application can sometimes have nuances that external folks may not be fully aware of and so having the packaging instructions (i.e. flake.nix) defined in the core repository can narrow that knowledge gap.
So, did Nix actually adopt flakes ? Because last time I checked, it was still an experimental feature that everyone insists on using, but… it's still experimental, which means you have to make effort to use it.
That’s my experience using Nix in a nutshell, every time I’ve tried to use it.
Flakes, home manager, half the packages I need, etc
It's a political thing, flakes are not going away or changing radically. Though the CLI commands aren't "stable" (guaranteed not to change wise). I read somewhere that someone of importance believes and thinks flakes will be enabled by default in 23.11 already.
The thing with nix is that features are experimental for a very long time, even though they are quite solid. It seems like it is mostly used so they can still make changes, like the CLI options or the names of attributes in the flake. The effort to use it is just adding extra line to the config.
So...devbox adopted flakes before Nix did? lol.
Does anyone already rely on LLMs way too much? Yes, but nix is a lot more predictable than LLMs are.
It's not for everyone, but when you work with infrastructure and can guarantee that the software the developers build is the exact same version you're running for them it feels good to have that conversation out of the way already.
Even after it passes the experimental stage, I doubt they'll drop support for the default install/config method using configuration.nix. Flakes will just another option but not required.
This probably goes against the flow, but I tried NixOS on a VPS and I found the tools to be inscrutable. Was so confused about Nix packaging and whether to use Flakes.
Depends on the person. I work in a company that uses Nix for everything one could possibly imagine, and I've found there are a few types of people:
1. The purist: Is on an actual crusade to promote Nix and stomp out any competition, because, "reproducible."
2. The enthusiast: Likes using Nix to speed up development but recognizes it has a ton of shortcomings, among others being not friendly to beginners, and tends to avoid using it in production critical stuff.
3. The ex-Nix: matches the enthusiasm of the purist in stomping out Nix wherever possible and actively promoting its demise.
I'm personally in the second category. I use Nixos for my main development machine and really like devshells. Beyond that, I try to avoid it as much as possible for production.
We use it for devshells, and it’s awesome. New devs install nix and direnv and they instantly have all the right versions of all of our tooling. A first day setup process is now done in minutes instead of a day. Flakes made it possible for us to package up internal and external tools and ensure consistency across a team.
I have no experience running it in production, but I imagine if you don’t want to use containers it’d be a pretty good option.
for devshells use https://devenv.sh
You can unify the purist (1) and the ex-nix (3) into a single label: the emotionally attached.
I'm kinda 1 & 2, sometimes one much more than the other.
I'd start by using Home Manager which is a way to use nix packages in a sensible manner on any distro.
If you like the properties of Nix, but find it confusing, you should check out Devbox! It simplifies the process of creating Nix-powered dev environments:
Flakes rule, impure eval drools.
But also, my "personal config" has two dozen imports and overrides nixpkgs on most of those. I have a list of complaints regarding flakes that is only dwarfed by 6+ year old general nix issues, but I could never go back.
the article does a good job of explaining why something like the flake system was necessary, but man oh man does the particular thing we wound up with have issues.
you can smell from a mile away that they were engineered to solve widely-experienced problems... as experienced by a single company, with an existing, idiosyncratic set of methodologies. and they just happened to get the blessing because eelco was at that company. unlike nix proper, however, where eelco had the entire internet for feedback, the core design of flakes ossified before the world at large had reason to care about them.
it doesn't help that nix's command-line ux is currently super splintered as a result, and while that will be ironed out in the long run, the thing in the name of which those tools got splintered is rather insulting.
i love nix but god damn this is the stage at which i'd take someone to couples counseling
I’m surprised all of these systems combine build logic and dependency graphs in the same config. It seems like flake might be a step in understanding these things are only tangentially related.
I would like to see composability of graphs (and other set operations on binary package repos) integrated into more dependency management systems.
FreeBSD is my server of choice and I’d love to say: create a package repo with my config package and it’s dependencies and nothing more and deploy from that knowing a million other dependencies can’t be pulled in (e.g. give me a new jail that pulls from subset).
It’s probably something I should prototype one day.
I don't fully understand what you are saying, and I don't know what systems you're including in "all".
The system you describe building is exactly what nix does, as well as debian, El, and Arch. Their spec files describe both build and runtime dependices, and installing the package does not install things like Make.
Nix goes a little bit further by only including runtime dependices that it can find it the build output. It does this by scanning the output files.
But I don't understand how you could separate build graph and runtime graph. If I declare something needs foo and bar, that is useless unless I can get built foo and bar. _Something_ has to know how to build the things this hypothetical system is installing.
RPM rspecs, port Makefiles, Maven, etc. do not just define package dependencies (build, runtime, etc.) and artifacts, they also have to intimately understand how to patch, build, and package the artifacts. Making the build and package system separate from the dependency management separates concerns and allows the ability for a completely declarative “dumb” dependency graph that can be reasoned about without dealing with build logic that must be executed.
This also allows build processes to evolve without affecting the public dependency graph. This makes it easier to show dependencies are modeled and exposed correctly and makes build logic private to consumers.
> allows the ability for a completely declarative “dumb” dependency graph that can be reasoned about without dealing with build logic that must be executed.
Nixer Domen Kozar called this property 'static metadata' in one of his talks¹ on Python packaging from a Nix perspective years ago.
The thing he was interested in was the ability to evaluate the dependencies of an upstream software package without having to actually 'install' it or evaluate bespoke upstream code to do so. The reason for interest in this within the Nix community is that by default Nix performs builds in a restricted sandbox, and one of its restrictions is that no network access is allowed.
To use upstream build tools (e.g., Maven or Cargo or NPM, etc.) inside the Nix sandbox, then, fetching dependencies and verifying their contents is deferred from the upstream build tool to Nix, which does so in a controlled, deterministic way that just fetches and doesn't have hooks to let those deps run custom code.
In order to make that happen, Nix has to 'know' ahead of time where to fetch those dependencies and what their contents will be, and what you describe wishing for here— a 'dumb dependency graph'— is more or less exactly what Nix wants to consume (although sometimes just a list of pairs of URIs and content hashes will do). For well-behaved upstream package managers, i.e., those which can emit comprehensive static metadata, that's exactly what Nix does: it just translates a Cargo or NPM lock file into its usual conventions for describing source archives, and then that can be used to download those dependencies in the usual safe/restricted way.
For package managers that don't emit adequate 'dumb' package metadata, Nix has to proceed by either emulating and replacing those upstream dependency resolvers (very error-prone) or by a hack implemented as a Maven plugin or similar that inspects dependencies as Maven resolves them and gets the metadata Nix needs.
As for flakes, flake inputs are certainly a kind of dependency that's distinct from package dependencies in the Nix world, and also 'dumber' in the sense that flakes don't have to be 'built' like packages to be consumed by other flakes. And yeah, in the case of flakes that provide packages, you can definitely swap one flake for another and allow that new flake to provide customized build instructions if those are needed for that version of the package it provides. But there are complexities and entanglements that flakes don't/can't eliminate, since downstream dependencies can still have implicit expectations of build outputs, and you can't really know in advance if the package in the new flake will meet all of those. You kinda still have to be able to peek into those and examine them and fix them up in a pinch. Plus you can write a flake in a way that depends on the build process of a package in a flake it consumes, e.g. by using package overrides on something provided in the upstream flake.
--
Yes!!
Awesome! I'm glad you found that context relevant. :)
It feels good to have your understanding of a problem and where different technologies might fit into better addressing it click into place like that. :D
I think the second point (nix flakes avoid 'stateful' channels) is the more compelling one. I don't recall an easy way for ensuring channels on different systems pointed to the same value; and flakes basically do what Niv did. And, it's much nicer to just install a flake, rather than having to add channels.
I'm glad to see the first point (flake.nix provides a consistent interface) mentioned. The consistent interface allows for the cli to be much nicer; `nix flake show` can list the outputs.
This is a bit of a unnecessarily provocative title, since the question of whether or not to promote the 'experimental' flakes system and replace the channels mechanism is still an active and controversial one in the Nix community. There hasn't been any significant news on this front recently; this article simply explains why flakes were introduced as an experimental feature.
Oh, it isn't intended to be provocative.
The title is simply literally the question that I was thinking about a few months ago when working on Devbox, and wrote a version of this post internally to answer the question for myself!
Ah, that's good to know :) It is just that I was expecting to read about some breaking news; that I missed that Nix deprecated channels or something! It is indeed a nice article explaining Flakes in general. Maybe "Why did Nix introduce Flakes?" might be a slightly better title for people coming with prior expectations like I did?
s/adopt/introduce
done :)
Thank you; that was quick! :) I apologise if my original comment came across as rather harsh!
idk, someone (moderator?) changed it back ...
I'm curious what you would change the title to so that you perceive it to be less provocative? In my (disclosure: not engaged in the community much at all) opinion the provocation here is simply because the topic touches on a controversial issue.
Maybe the "Why did ...?" can come across as confrontational. -- If something's good, you're less likely to ask "why do that?".
"What Nix Flakes Solve" or "Benefits of Nix Flakes" would avoid that.
But, I didn't read the title as provocative.
I find the default way to work with nixos to be a lot easier to use compared to flakes. I am sure flakes are great and all but as with everything nix you can't just integrate it into your normal flow, you have to jump all in with it. I don't think that is a positive.
It is possible to use flakes on systems otherwise managed with channels. Just enable the "nix-command flakes" experimental features[1] and stuff like "nix run" will work.
[1]: https://nixos.wiki/wiki/Flakes section "Enable Flakes".
> I am sure flakes are great and all but as with everything nix you can't just integrate it into your normal flow, you have to jump all in with it.
AFAIR, the only irreversible impact is switching from `nix-env --install` to `nix profile install`.
With repositories, you can add a flake.nix, and still use nix-build or nix-shell however you did before. e.g. You can have the flake.nix import the default.nix or shell.nix code.
Pretty much, still not sold on flakes. My nixpkgs is also git repo not channel
Am I the only one completely confused by the examples that appear to put command line arguments after shell end-of-line comments? Is that not a typical shell?
example:
nix run .#cowsay -- flakes are neat
yikes! thanks for pointing that out. The blog's renderer is confused too and renders the `#cowsay -- flakes are neat` as a comment.
To explain what's happening:
- The example above is running `nix run <flake output attribute> -- <arguments to pass to the flake output's binary>`.
- The `<flake output attribute>` here is `.#cowsay`, which is to be read as: `<flake reference>#<attribute path>`. The # is a separator here.
- The `<flake reference>` being `.` implies its a local flake at the current directory. The `<attribute path>` in this case is the output from the flake i.e. the `cowsay` program.
further fun to be found at: https://nixos.org/manual/nix/stable/command-ref/new-cli/nix....
As a meta-comment, yeah, this is kinda non-trivial for most of us. This is why we build devbox which provides a more familiar UX like `devbox add cowsay && devbox run -- cowsay "flakes are neat"`
"--" isn't anything special in a typical shell. It's just another argument, one that customarily means "pass the rest of arguments on to a subprocess". Or are you confused about the "#"? It doesn't start a comment in the middle of a token.
> one that customarily means "pass the rest of arguments on to a subprocess".
It's convenient for passing options and args to a subprocess, but it customarily means "don't interpret anything following this as a short or long option, just as a positional argument". For example:
touch -f # error touch -- -f # creates a file named "-f" rm * # oh crap we just passed "rm -f" a bunch of files rm -- * # don't treat that "-f" as an option, just delete the file named "-f"It was the "#". I didn't realize that about the behavior in the middle of a token, and apparently the syntax colorizer doesn't either.
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...
The first -- argument that is not an option-argument should be accepted as a delimiter indicating the end of options. Any following arguments should be treated as operands, even if they begin with the '-' character.
The comment syntax for most (virtually all?) shells actually doesn't start with a pound sign but with the word boundary or whitespace preceding the pound sign. You can have unescaped pound signs within tokens in shell languages, and thus they are used (perhaps unfortunately) as delimiters in the current flake URI schema.
It's nix run, the dot means current directory, # means flake, cowsay is the attribute, and then anything after -- is passed to what's run by that command, which is typical of many Unix programs.
So it's run the cowsay attribute from the flake in the current directory, passing flakes are neat as input to cowsay.
I prefer Flakes non-Flakes, but it’s disappointing that you can’t specify mirrors unlike the fetch* commands. Things go down on the internet & mirrors are a way to cover that sitution. There’s nothing like a failing CI because the one of the origin’s servers are down.
I thought that this option solved this problem? https://nixos.org/manual/nix/stable/command-ref/conf-file.ht...
But with the myriad of fetch* functions maybe it only applies to some.