Building packages at scale
perkin.org.ukInteresting article with historical context that goes 10 years back and many detailed insights!
Using pkgsrc at our company since a bit over a year has been excellent thanks to the great work from jperkin and many others. It is one of the tools i wish i had known earlier about since it makes life sooo much easier.
- Quartly stable releases mean that you have recent software most of the time.
- It is available on many platforms (OS X, Linux, SmartOS/Illumos, the BSDs, ...) so you can use the same packages on all your systems.
- Very easy to build binary packages of custom software / custom options / different versions / add your own patches. This is critical for most sysadmins/ops people - and if you ever tried to setup koji for CentOS you will love the simplicity of pkgsrc...
In my opinion pkgsrc is the most underappreciated "devops"-tool out there. It gave us much more control over our software stack and reduced the amount of work we had to put in to archive our goals.
I also expect the effort to reduce bulk-build times to further improve the software quality of all things in pkgsrc. Instead of weekly/daily bulk-builds (depending on the platform) it will be possible to have 4 or more reports a day. This quick feedback will make determining if/which commit broke/fixed something more immediate. Great stuff!
Great article, impressive attention to detail. It sounds like a situation that would benefit from a compiler cache, such as the one found at https://ccache.samba.org/. Is there any benefit to be had there or is the storage tradeoff too expensive?
I want to try ccache at some point (with pkgsrc it's trivial to add), but I'm not convinced it will help.
Given the distributed nature the first implementation would keep the ccache objects on NFS, and that's going to cause additional latency for every cc invocation to the point where even a cache hit will likely be slower.
The next step would be to synchronise all of the ccache objects back to each build zone and then loopback-mount the directory into each chroot. This may provide a small increase in performance, but at the cost of additional complexity. I also wonder whether it would be enough of a performance win compared to the extra time it will take to rsync the objects to each build zone at the start of the build.
I will likely do this at some point anyway, as I'd like to have some hard numbers to back up my theory, and if it turns out to be a win, even better ;)
I've built a few such configurations myself, but never again. The worst behavior is that sometimes, the NFS caches result in compiling old versions of some files some of the time. When you're rapidly tweaking things for debugging and sometimes some changes only make it into some compilation units, it can drive you insane.
And it doesn't go much faster than just compiling locally with Clang on an 8 core CPUs with SSDs.
You could have a separate ccache directory for each package, such that you can enable/disable using ccache on a per-package basis. Then you can rsync the objects in parallel with the build process, but only for the 'long tail' packages (kde, etc), such that the ccache objects are available by the time that the those packages start building. Definitely adds complexity though.