Rejuvenating Autoconf
lwn.net> if your customers need to be able to build a shared library on Solaris, AIX, HP-UX, IRIX, and all the BSDs
Are there even any notable commercial projects that use autotools?
Autoconf solves problems that existed in the 90s unix market which are just not very interesting or relevant today. For a few decades now "portable" has meant linux, windows and mac.
A lot of effort is spent on generating shell scripts for the lowest common denominator and this is both useless (nobody cares about 90s unix, everybody can install bash) and insufficient (bourne shell is not native on windows).
Perhaps energy would be better spend on migrating everybody towards newer build systems or just plain makefiles. A lot of projects have migrated already, does anyone have statistics about what percentage of a distro like Debian needs autotools?
Also everyone uses package managers these days, and if you ever need to install a package manager manually, spending 2 seconds to specify the location of a certain library, or googling “cannot find -lpng”, doesn’t take much effort. The automatics of library finding is super complex and can really get in the way of package/repo maintainers. Today, it’s probably a better idea to optimise build systems to be used by people who understand how the relevant programming language finds libraries, and not to “help end users understand how to compile their code”.
Autoconf often does a good job in having some “—with-libX11=some-path” but sometimes people just do weird stuff with their packages.
But maybe I’m just butthurt from trying to package stuff to nixos and having too many “smart build tools” fail.
While we're here, can we please also get rid of libtool?
Folding that into Automake is under, well, fairly active or at least repeated discussion. :)
> Are there even any notable commercial projects that use autotools?
Did you even read the article? The work is supported by Bloomberg for their internal use with Python, a clear commercial usage. https://lwn.net/Articles/835687/
Bloomberg has about 10 G$/y in revenue. https://en.wikipedia.org/wiki/Bloomberg_LP
> For a few decades now "portable" has meant linux, windows and mac.
That’s a great way to stifle any kind of innovation. Some newcomers may still appear that are worthwhile.
There are likely to be some, but they will be a minority. And just like open source projects, their number will be dwindling. In both arenas, I've encountered CMake more than anything else over the course of the last decade.
You're absolutely correct that Autotools is no longer synonymous with portability. If it ever was. Not catering for Windows or iOS means it's missing out on the lion's share of the market. There's a massive misplaced priority in catering for 90s Unix platforms which have marginal use by hobbyists and zero commercial use, above current open source and proprietary platforms. And this doesn't just extend to operating systems, it also extends to language support. C++17, C++20, Rust, Go, Swift, you name it. It's not kept up. Likewise build systems. It's make-only. And what about language and platform feature tests and support for stuff like threading, locking etc. It's non existent other than some third-party support which is quite limited.
It's for all these reasons, as well as other well documented problems, that I moved all my personal projects over to CMake, as well as work projects, and other open source projects. I needed to support Windows. I needed to support building libraries on multiple platforms. I needed threading support. I needed C++14, and C++17 support.
For some open source projects I maintain both Autotools and CMake support. I've had several instances of people trying to build on e.g. Solaris and having problems. I always ask "Were you using the Autotools build, or CMake?" They tell me they used the Autotools and I suggest retrying with CMake. It always works. In short: portability to other Unix systems is inferior today compared with CMake. That's supposed to be its main advantage...
Agreed about the shell. There is no point to M4 and it could be removed compatibly by replacement with shell functions. This was true a decade ago. It was even true two decades ago.
With regards to migration. The time, money and effort required to upgrade the Autotools would be better spent on converting the affected project(s) to use CMake or some other more modern build tool. The functionality gap is simply too big to overcome at this point, and the contributor base for CMake is large. The FSF have only themselves to blame for their contribution and development practices hindering forward progress, as well as a lack of concern for the real-world needs for software development. The turnaround time for submitting a CMake change is a few days to get it change reviewed and integrated, and a few weeks to be in a release.
I've converted some large and complex projects to use CMake over the years, including Xerces-C, Xalan-C, libtiff and others. All in all, that time was well spent, and I'd rather have done this than waste time updating Autoconf/make/libtool.
Regarding Debian. Current unstable Build-Depends is 1999 (CMake), 1544 (Autotools), 395 (Meson). So I think we're a good way past the tipping point now.
Full disclosure: I've done the FSF copyright assignment for Autoconf and Automake and submitted some major features like C99 support. I'm also a CMake maintainer and I've submitted many different changes there. Primarily lots of Find module support including the Boost support, but also some lower-level bits and pieces as well. The community surrounding CMake is very active and also very friendly to submissions from new people. Kitware really deserve appreciation and thanks for their fostering and growing the community and committing resources to it including timely and friendly code review.
> Current unstable Build-Depends is 1999 (CMake), 1544 (Autotools), 395 (Meson)
These days pretty much everything build-depends indirectly on autotools, via debhelper → dh-autoreconf → autoconf.
These numbers are probably more meaningful:
$ wget -q https://deb.debian.org/debian/dists/unstable/main/Contents-source.gz $ zgrep -E $'(^|/)configure[.](in|ac)\t' Contents-source.gz | cut -f 2 | tr ',' '\n' | sort -u | wc -l 5562 $ zgrep -E $'(^|/)CMakeLists[.]txt\t' Contents-source.gz | cut -f 2 | tr ',' '\n' | sort -u | wc -l 2455 $ zgrep -E $'(^|/)meson[.]build\t' Contents-source.gz | cut -f 2 | tr ',' '\n' | sort -u | wc -l 466> There is no point to M4 and it could be removed compatibly by replacement with shell functions.
Autoconf has been using shell functions since 2008 (support added by me :)). M4 is used to properly handle dependencies between tests and to store information that is used later when generating files.
> In short: portability to other Unix systems is inferior today compared with CMake.
CMake requires a C++11 compiler and quite a few C libraries like libuv, so you can't even compile it on platforms like HP-UX.
Brad has submitted generic POSIX support to libuv for fallback support (I forget which platforms exactly were in consideration at the time). We even have a tracking issue for HP-UX support which tracks the status:
https://gitlab.kitware.com/cmake/cmake/-/issues/17137
Part of it is HP saying they'd have a C++11 compiler and then not, some is modern GCC being a problem to compile as well.
C++11 and libuv have vastly improved the situation in CMake's codebase itself and letting HPE hold everyone else hostage for that would not have been worth it (and it takes a lot for us to drop platforms). IRIX was dropped because its owner killed it off and we're looking at dropping support for macOS 10.9 and older (in premade binaries; feel free to compile yourself) because we don't actually have tests running on those older releases.
https://gitlab.kitware.com/cmake/cmake/-/issues/20825
Full disclosure: CMake developer here.
> primarily lots of Find module support including the Boost support, but also some lower-level bits and pieces as well.
Last time I checked, the CMake docs say that adding find module support is responsibility of the user. Which is a main reason why pkgconf-based dependency resolution works much better with other build systems. If you compile package abc on, say, Ubuntu, and it needs package xyz you do not need to bother about how it is found. You just install xyz.
What I want to say, CMake makes things way more complex and solves problems which exist only on Windows. Otherwise, it does not add much, and is a burden. Why should developers on Linux carry that burden?
> Not catering for Windows or iOS means it's missing out on the lion's share of the market.
Do you mean the market for free software?
I guess the GNU/Linux developers are not that keen to work for free to support commercial, nonfree platforms.....
Most of the GNU software catalogue has historically been supported on nonfree platforms. Nothing to do with Linux here, it's not part of the equation.
None of this consideration is really concerning free software or proprietary software specifically. It's simply looking at cross-platform capability. If you need to develop cross-platform software of either sort, it's not typical to be able to ignore Windows. It's ~90% of the market. Similar considerations for other markets like smartphones.
None of that is reflective of my personal preferences, simply stating the requirements which drive the build system considerations of many projects. The GNU Autotools are unsuitable for the vast majority of current systems in existence, and that means selecting it for its portability is nowadays often not a possibility because their choice to not support common platforms makes it untenable.
This is like complaining that the supply chain of oat milk is incompatible with 90 % of the systems that produce (cow-based) butter. We don't necessarily care.
> And this doesn't just extend to operating systems, it also extends to language support. C++17, C++20, Rust, Go, Swift, you name it. It's not kept up.
Guix does support, for example, Rust and other languages relevant for Linux pretty well, and in a much better defined way than CMake.
Poul-Henning Kamp's article, A Generation Lost in the Bazaar, explained the absurdity of autoconf & libtool at length. I quote some of it, below.
“At the bottom of the food chain, so to speak, is libtool, which tries to hide the fact that there is no standardized way to build a shared library in Unix. …the source code for devel/libtool weighs in at 414,740 lines. Half that line count is test cases… the tests elaborately explore the functionality of the complex solution for a problem that should not exist in the first place. Even more maddening is that 31,085 of those lines are in a single unreadably ugly shell script called configure. The idea is that the configure script performs approximately 200 automated tests, so that the user is not burdened with configuring libtool manually. This is a horribly bad idea, already much criticized back in the 1980s when it appeared, as it allows source code to pretend to be portable behind the veneer of the configure script, rather than actually having the quality of portability to begin with. It is a travesty that the configure idea survived. …
“The configure scripts back then were written by hand and did things like figure out if this was most like a BSD- or a SysV-style Unix, and then copied one or the other Makefile and maybe also a .h file into place. Later the configure scripts became more ambitious, and … rather than standardize Unix to eliminate the need for them, somebody wrote a program, autoconf, to write the configure scripts.
“Today's Unix/Posix-like operating systems, even including IBM's z/OS mainframe version, as seen with 1980 eyes are identical; yet the 31,085 lines of configure for libtool still check if <sys/stat.h> and <stdlib.h> exist, even though the Unixen, which lacked them, had neither sufficient memory to execute libtool nor disks big enough for its 16-MB source code.”
From https://twitter.com/brainwane/status/1319692088758489089 :
For most of this year, @elwoz and I have been working on a pretty ambitious project:
* to make a fresh release of GNU Autoconf, a crucial #FLOSS build tool that hadn't had a new release since 2012
* to get paid for that
* to help put Autoconf on a more sustainable footing, so it doesn't have to get rescued again a little while down the road
Use αcτµαlly pδrταblε εxεcµταblε it solves problems that actually exist in this day and age like being able to build once run anywhere TUI apps for Windows/Mac/Linux/BSD. See https://github.com/jart/cosmopolitan/commit/9e3e985ae5fdccc9... and https://justine.storage.googleapis.com/ape.html
Nobody will use a tool that isn't even compatible with their keyboard.