Lua and Python

Did you know...?

LWN.net is a subscriber-supported publication; we rely on subscribers to keep the entire operation going. Please help out by buying a subscription and keeping LWN on the net.

From a high-level perspective, Lua and Python are similar languages; both are "scripting" languages that are compiled into bytecode instructions that run on a virtual machine. But the focus of Lua has generally been toward embedding the language into some larger application or system, rather than as an alternative for, say, Python, Perl, or Ruby as a general-purpose language. That is not to say that Lua is not capable of handling any of the tasks those other languages do, but that it has not really been the target, seemingly. Some recent discussions in the Lua community have explored possible changes in that regard, particularly around the idea of providing a larger, richer standard library.

In mid-December, Gavin Holt posted a message to the lua-l mailing list noting his overall happiness with the language, but observing that support for the "batteries" was lagging. The "batteries" term presumably comes from Python, which has long had a "batteries included" philosophy—it ships with a large standard library that is meant to cover many of the tasks a user might encounter, without needing to install additional packages. As Lua has progressed through 5.1, 5.2, and 5.3 (with 5.4 on the horizon), however, some of the packages have not been updated, so Holt cannot use them with recent releases of the language. Many of the Lua libraries are written in C, which need to be compiled and distributed as binaries for some users.

Lua is well suited to the "non-compiling user", as are many other interpreted languages. The popularity of these scripted languages is in part the ability to use them without learning/maintaining a compiler. However, there is a ceiling to what can be achieved with Lua alone and from that point onwards you need compiled libraries aka "Batteries" (lfs, winapi, rex_pcre, clipboard, afx, lpeg, hunspell, lsqlite, vcl, gslshell). I am very grateful to those who have made compiled libraries for windows available to download, I don't have or want a compiler!

As might be guessed from the last point, Holt is running on Windows—Linux users may be somewhat more accustomed to having and using a compiler on their systems. "I know the 'non-compiling' user may not be the main target audience for Lua, but as a beautiful interpreted language I feel the lack of 'Live Batteries' prevents Lua becoming as popular as it should." He wondered if there was any effort to put together a distribution of pre-compiled libraries.

There were several suggestions of places to look for the pieces that Holt needs in the thread, but the overall picture is that Lua lacks a standardized set of libraries—which has both good and bad points. Dael Vnaja thought it would be useful to have "an alternative Lua distribution/fork that includes these 'batteries' from the language itself and not from 3rd party libraries", but that the user base for Lua might be too small to sustain one. Coda Highland said that the actual user base was quite large, but that each of the big players has its own requirements: "Everyone has different needs so the places where serious money is getting poured into using Lua don't really have much of an incentive to standardize -- it wouldn't give them any benefit."

Choosing libraries

But there are some problems trying to choose reasonable libraries for Lua, Steve Litt said. First, options need to be found, then they must be installed and tried, which may require multiple iterations. If the library is not packaged by the distribution being used, it needs to be grabbed from elsewhere (possibly using the LuaRocks package manager) and any users of the code need to also get instructions on how to install the library. It is quite different in the Python world:

He suggested that "an official list of approved Lua libraries", which could be searched and that would provide information on the strengths and weaknesses of the library, would be a good place to start. It would be far better than "having to ask online and getting lots of opinions".

Some agreed with that idea, but others were not so sure. Python exists, so perhaps those who need a full-featured standard library should turn to that, Dibyendu Majumdar said. "Lua's strength is lack of any baggage, which makes it better for embedded use." But no one is suggesting that a set of batteries become a requirement for Lua—even the existing Lua standard library can be compiled out for use cases where it is not needed, Lorenzo Donati said. The main reason that Python is a better general-purpose language than Lua is "because of its libraries", he continued.

By all accounts, Lua works quite well in its niche, but some see the potential for it to grow beyond that niche—and the main thing holding it back is the lack of a rich, standard library. As Litt put it:

But given the fact that Lua, as it has existed for the past five years, is a pretty much perfect language, curating a group of approved add-ons would be the best way to improve Lua.

The conversation picked back up again in mid-January when Donati posted some links about the popularity of Python. Stefan Lederer replied that it is not surprising that Python is as popular as it is "because of powerful libraries and easy scripting", but that does not mean the Lua community should be trying to create a "full-fledged Better Version Of Python That Is Actually Lua". He thinks that Python suffers from problems on multicore systems that are only going to become more acute and believes that Lua is better positioned for multicore. That needs more work too, however.

Litt suggested that the existence of its batteries made Python a better choice for a larger project: "If you start a project in Python, you're more likely to finish it in Python as opposed to if you'd started it in Lua and expect to finish it in Lua." He posited two possible directions for Lua, either recognizing that its lack of batteries is holding it back for the general-purpose use case or simply deciding that Python already fills the other role, so Lua should stick with the embedded use case. He was not in favor of the latter:

But a set of standard libraries has its own set of downsides, Peter Hickman said. Languages all have their sets of tradeoffs; Lua is simple and fast, even though it sometimes lacks for a particular library he might need or want.

Although I get pissed when searching for an existing Lua library to do some task to find either zero or six. Not having standard libraries is not necessarily a bad thing. In other languages the standard / default library for something tends to have a kitchen sink set of features just in case someone might need feature X. This comes at a cost, the codebase is larger, it has codepaths that are addressing features you have no interest in, hitting performance and eating memory. More code means more possible bugs. Just look at the standard Python logger. It is truly a marvel of engineering. Of which I use at most 10% but the remaining 90% is baggage my application must carry. Same with Ruby, every unused language feature is hitting performance and eating memory.

Collaboration platform

Oliver Schmidt thought that the missing piece is not really a set of batteries, but is instead a way to collaborate on gathering, rating, and discussing the existing (or new) libraries for Lua. In essence, there is a lack of a Lua community working together to create a code repository that is "useful for newcomers and common use cases". In order for Lua to grow, there is another path should perhaps be considered:

Everyone cooks his own soup that perfectly fits his use case. This is perfectly OK IMHO, this is not a "bad thing", but everyone who wants Lua to have more community impact sees that there is something missing in the current way.

Currently, the LuaRocks site has a bunch of libraries that can be installed, but some important ones are missing and the site lacks many of the features Schmidt thinks are needed. One way forward might be to extend the LuaRocks site, but perhaps setting up a new site on an existing platform (e.g. GitHub) would be an alternative, he said. As became clear in the rest of that thread, though, the community effort required for any kind of solution may not really be available.

Blessing

There is also the question of which libraries would go into some putative richer standard library. Some in the threads seem to think that the Lua team should "bless" a set of libraries, though that has problems of its own. As one of Lua's founders and a member of said team, Roberto Ierusalimschy, put it: "I would break the steps even further: first step is a stable library. Nobody can bless anything unstable."

Another sub-thread took up the idea of a separate repository where the community parks its library code to be improved upon collaboratively; Rodrigo Azevedo proposed the LuaDEAL (for "Lua DEad Alive Libraries") project for creating a kind of standard library. A first step is to identify which libraries fit: "It means, when someone on this list asks about a library, everybody must agree on a first default answer. Public libraries are blessed by the community, not a specific restricted set of people." The choices are not meant to be the "best", just something that is useful to get a particular job done. In addition: "They must have a well defined and stable 'Lua-ish' API. Documentation is mandatory."

There were some discussions about what that API should look like, and even a fair amount of agreement, but the fact remains that the Lua community is small. Dennis Fischer split up Lua users into a handful of categories and noted that Lua enthusiasts would seem to be the smallest of those groups. Growing the numbers of enthusiasts is needed, but there are some problems in the way of doing so:

So the alternative would be to somehow capture more of the users that really only pick up Lua for some very specific task. The problem with this is that, sadly, many Lua APIs are just trash. There's a world of globals, ad-hoc code and poorly translated pythonisms out there, and not many examples of idiomatic, well-designed APIs to offset it all.

That seems to be the crux of the issue: there is no collection of Lua libraries that "everyone" could coalesce around, even if enough everyones could be found. There are tons of Lua libraries available, but finding the "right" one is difficult, such that many users simply go off and implement their own. Even if a library appears to be suitable, it can be hard to find out if will work on the operating system of interest or the Lua version (or alternate implementation such as LuaJIT) needed.

That stands in stark contrast to the Python world, where its large standard library is immediately available to import. Beyond that, add-on libraries are plentiful and generally well-specified on the Python Package Index (PyPI), which can be installed via pip. To a large extent, the idea of what is "Pythonic" is fairly well-understood in the community as well; "Lua-ish" seems to be a bit more amorphous, or at least not well socialized within the community.

Certainly Python has struggled at times with its batteries. Some modules were perhaps mistakenly adopted into the standard library, others have not aged well, and some have even bit-rotted to an extent, which has led the project to discuss removing some "dead" batteries. There are also a fair number of API warts among those modules. But overall, the choice made early on to include a standard library with the language itself has paid big dividends for Python—and its popularity.

Lua is not much younger than Python—Lua came about in 1993, while Python was started in 1989—but Lua's focus has always been different. Trying to change the inertia of the project and its community nearly 30 years down the road is likely to be a difficult task. It is not insurmountable, perhaps, but it will take a great deal of focused effort—one library at a time. Whether the effort is worth the price is hard to judge in the early going as well.

Nothing here is meant to disparage Lua or its community in any way. By all accounts, Lua is an excellent choice for a lot of tasks. Its community seems friendly and welcoming as well—in many ways lua-l is reminiscent of the Python discussion forums. The lack of batteries is not really a new topic in the community, and a variety of solutions have been tried, so it will be interesting to see where things go from here.

Index entries for this article
Python	Other languages

---