We're bad at marketingWe can admit it, marketing is not our strong suit. Our strength is writing the kind of articles that developers, administrators, and free-software supporters depend on to know what is going on in the Linux world. Please subscribe today to help us keep doing that, and so we don’t have to get good at marketing.
GNU Autoconf, a widely used build tool that shines at compatibility with a variety of Unixes, has accumulated many improvements since its last release in 2012 — and there are patches awaiting review. While many projects have switched to other build systems, interest in Autoconf remains. Now, a small team (disclaimer: including me) is rejuvenating it, working through some deferred maintenance and code review. A testable beta is now out, a new stable release is due in early November, and interested parties can build on this momentum to further refresh the rest of the GNU Build System (also known as Autotools).
A widely used default
GNU Autoconf is a tool for producing configure scripts for building, installing and packaging software on POSIX systems. It is a core component of the GNU Build System. When a user installs a software package on the command line by compiling it from source, they are often instructed to run:
$ ./configure; make; make install
Those steps do the following:
- configure: test system features with attention to portability concerns, prepare and generate appropriate files (including a makefile) and directories, etc.
- make: use the makefile as instructions to build the package, performing any necessary compilation steps
- make install: place the built binary and any other needed files into the appropriate location
configure is a portable shell script that must run on many platforms. Writing a configure script by hand can be tedious and difficult, so Autoconf helps automate this process. A software developer writes a configure.ac file specifying system features the software will need (e.g. "is the X Window System installed, and if so, where?"). Each test for a system feature is a macro written in the GNU M4 language. Autoconf comes with many macros that developers will likely need, and a library of add-on macros ("autoconf-archive") (source) provides dozens more.
Thus, in the base case, a programmer wanting to distribute code to be built with the GNU Build System needs to write only a bit of M4 in configure.ac, and would likely only need to use one or two additional macros from autoconf-archive. They do need to learn more M4 if they need configure to detect a system feature for which there is not an existing macro.
Autoconf has built-in support for various compiled languages: C, C++, Objective C, Objective C++, Fortran, Erlang, and Go. More crucially, it performs feature detection with knowledge of a wide variety of POSIX platforms. If you are building new software that has few arcane dependencies and your users are all on modern Linuxes plus FreeBSD, or if you want to make Ninja build files, perhaps you'd be better served using alternatives such as CMake, SCons, or Meson — and indeed many projects have switched away from the GNU Build System over the years, including GTK+ and KDE. Complaints that the GNU Build System is slow, complex, and hard to use have been aired (including in LWN's comment threads) for years. However, if your customers need to be able to build a shared library on Solaris, AIX, HP-UX, IRIX, and all the BSDs, then Autoconf will come in handy.
From 1991 to the present
Autoconf's founding in 1991 and immediate subsequent history is chronicled in its manual and in the book The GNU Project Build Tools. Its function in the 1990s and early 2000s was to smooth over differences among the proliferating Unix variants. Autoconf's last big change was the version jump from 2.13 to 2.50 in 2001, which broke many existing configure scripts and required several follow-up point releases. Version 2.50 extensively overhauled several components, including autoupdate, and changed cross-compilation defaults; it was such a disruptive release that some users are still using 2.13 so as not to have to port their old scripts.
However, in recent years, Autoconf's star has faded. Linux's ascendance has made it easier for developers to get away with ignoring portability among Unixes — and the GNU Build System's balky Windows integration doesn't help those who need to deliver software to all three major desktop operating systems. But older, more complex projects include legacy code that already depends on Autoconf; converting it would be risky and expensive. In addition, competing build systems don't cover all of the edge cases that Autoconf does.
The rise of languages that use their own package management (such as Python, Perl, Go, and Rust) means that developers writing single-language code bases can avoid system-level build tools entirely. On the other hand, if you're writing software that combines C++, Fortran, Python, Perl, and Erlang, the GNU Build System can make those multiple languages play well together. It is more language-independent than, say, setup.py, and you can use the built-in macros plus the autoconf-archive macros to say: "I need to be able to use the 2011 dialect of C++, and I need this particular Python module installed".
Users of the GNU Build System need stability, multi-language
compilation, and cross-language compatibility, so the incremental
improvements and bug fixes in post-2.50 versions of Autoconf have supported
those goals. Autoconf's users have lived with version
2.69 since 2012; there have been no stable releases since
then. However, development has not stopped; commits to the Git
repository continued. Users also submit patches using the
autoconf-patches mailing list and Savannah; by our
estimation, as of mid-2020, there were hundreds of these patches awaiting
review. (There are fewer now, but we'll get to that.) Maintainer Eric Blake
had
been aiming to make a release but hasn't had time; as he said
in 2016: "The biggest time sink is digging through the mail archives to
ensure that all posted patches that are still relevant have been applied
".
Fresh momentum and work in progress
My involvement in Autoconf started when Keith
Bostic emailed the autoconf mailing list in January, asking: "is there
someone we could pay to do a new release of the autoconf toolset?
" Zack
Weinberg, an Autoconf contributor, forwarded the note to me.
Bostic was interested in Autoconf's future because one of his projects used it. He funded Weinberg and me to assess the work remaining; as we did that, we talked with Autoconf's maintainers (including Blake and Paul Eggert) and they agreed that we could do further release work. Then, starting a few months ago, Bostic — along with Bloomberg and the GNU Toolchain Fund of the FSF — has further funded our work so that we can work toward a 2.70 release in early November.
Weinberg released a testable beta version in July (even though this is a beta version of 2.70, the beta is labeled 2.69b) and a second beta, 2.69c, in September. We are now partway through our goals for this funded project:
- Along with other users, we've started testing the upcoming release against real Autoconf scripts for complex projects, but haven't yet put it through its paces with Emacs, GCC, and CPython.
- Since Autoconf has no continuous integration (CI) at present, we're going to set up proper CI system to find regressions, at least on Linux, probably at sourcehut.
- We've gotten a fraction of the hundreds of disorganized patches and bug reports filed, so Autoconf contributors can prioritize and assess our backlog; unfortunately, we don't have enough time to organize even half of them.
- We've reviewed several high-priority patches that downstream redistributors (such as Arch Linux and the Yocto Project) already carry and merged them into the mainline repository.
- We've started working with existing maintainers, contributors, and users to get the project on a more sustainable path.
These activities, fortunately, have gathered more momentum with testing and review help from existing maintainers and contributors, plus new volunteers. And the new scrutiny and testing have also led to fixes in related tools, such as the portability library Gnulib.
Speedups, bugfixes, and stricter parsing
The 2.70 release notes/NEWS file, which is in progress at the time of this writing, discusses speedups, several small new features, and many bug fixes that will be in the new release. The bug fixes alone are an appealing reason to upgrade. For instance, configure scripts generated by the new Autoconf will work correctly with the current generation of C and C++ compilers, and their contents no longer depend on anything outside the source tree (this is important for build reproducibility).
2.70 does, unfortunately, include a few places where backward compatibility with 2.69 could not be preserved. In particular, Autoconf is now more strict about M4 quotation syntax (a perennial headache for Autoconf users) and some macros do not perform as many checks as they used to, which speeds up the configuration process but can break configure scripts that assumed that some shell variable was set without actually calling the macro that sets it. In addition, more configure scripts now require the helper scripts config.sub, config.guess, and install-sh. (See the release notes for the complete list.)
Maintainers of complex Autoconf scripts will find it well worth their time to test the beta releases and report any problems encountered to the Autoconf mailing list.
Beyond this release: future resilience
In October and early November, Weinberg and I will likely use up the last of the funding we received. We intend to solicit more funding and to get more corporate contributors to commit to helping with testing, code review, and so on. After all, a big open question is: who will commit to serving as release manager for Autoconf 2.71? It might make sense to schedule that release for around 12-18 months from now. After Autoconf 2.50, a steady stream of people reported problems that contributors fixed in the next several releases. If we have someone motivated to triage bugs and prioritize and review patches, it may make sense to do that again, especially since, after 2.70, there will almost certainly be new bug reports, including for bugs introduced by the release but not found during beta testing.
There's also an open question as to who will work to organize the multiple backlogs of patches and bug reports, so that maintainers can properly assess, prioritize, and delegate work. Even once we finish the work that we've already received funds to perform, there will still remain scores of patches languishing in the various mailing lists and/or in patch sets currently carried by distributions (such as OpenEmbedded and the BSDs) but not yet merged back into the mainline. Getting all of those into Savannah, or the new GNU forge when that shows up, would help contributors, as would proper CI on multiple operating systems and environments. Gathering all of the submitted patches into one forge will also help downstream distributors cherry-pick specific fixes to carry in between Autoconf releases.
Autoconf has only been able to revive itself because of the funding from our sponsors. Conversations in the coming months will reveal whether and to what extent they and other enterprise users want to invest to keep Autoconf on a stable footing. This is certainly not the only piece of old software that free software depends on as infrastructure and that has significant deferred maintenance that needs doing; there are closely related projects that could also stand to be revitalized. Automake is one example; Libtool could be deprecated, and have its features refactored into the faster and more integrated functionality in Automake.
Regardless, in this case, it has been gratifying to help break a bottleneck so that users of a widely used, even crucial part of the open-source ecology can benefit from eight years' worth of improvements — and get Autoconf in better shape to make future release cycles better too.
[I would like to thank Zack Weinberg for reviewing this article.]
| Index entries for this article | |
|---|---|
| GuestArticles | Harihareswara, Sumana |