Why Create a New Unix Shell?

| blog | oilshell.org

2018-01-28 (Last updated 2019-10-05)

Whenever I announce a new Oil release, some readers are confused by the project.

This post, which I'll update periodically, explains the project's motivation from several perspectives. Because Unix shell is an old and successful technology, there are many ways of looking at it.

Introduction

Before explaining why I created Oil, let's review what it is. You can think of a Unix shell in two ways:

1. As a text-based user interface. You communicate with the operating system by typing commands.
2. As a language. It has variables, functions, and loops. Shell programs are text files that start with #!/bin/sh.

In this document, we'll think of Unix shells as languages. The Oil project actually has two languages: OSH and Oil. Let's define these terms, along with two others for context:

• POSIX shell is a language defined by a standards body. For example, Debian Linux and FreeBSD are both Unix operating systems that provide POSIX shells. Nearly all shells are POSIX-compatible: bash, dash, ksh, etc.
• GNU bash is the most popular shell implementation in the world. It was first released in 1989, and implements the POSIX standard plus many extensions. It's the default shell on most Linux distributions, runs on BSD Unix variants, ships on Mac OS X, and now runs on Windows.
• OSH is a new shell implementation that's part of the Oil project. It's compatible with both POSIX and bash. The goal is to run existing shell scripts. As of January 2018, I've made major progress toward this goal.
• The Oil language is a brand new, incompatible, shell language. The idea is to fix more than four decades of accumulated warts in the Unix shell. Many Unix users are angry that shell is so difficult, and Oil aims to fix that. (Example: why is x=1 different than x = 1?)

More about the Oil language:

• I've designed it, but not yet implemented it. However, the OSH-to-Oil translator works.
• Compared to POSIX shell and bash, it should be easier to learn, write, and debug. It has more consistent syntax and semantics.
• Oil will do everything bash can, and more. For example, it will have better structured data types.

You can get a glimpse of Oil at the end of this document. I link to several blog posts; the ones that describe the Oil language most clearly are the two on Translating Shell to Oil.

In summary, as I wrote in Roadmap #5:

Oil is your upgrade path from bash. It's the only language that shell / bash can be automatically translated to.

Update 10/2019: Oil is no longer focused on auto-translation, but it's still your upgrade path from bash. See You Can Now Try the Oil Language for up-to-date information.

If you're still confused after reading the rest of this post, leave a comment. I'll answer your question, and perhaps update this post with this answer.

Frequently Asked Questions

This section paraphrases questions I've received and summarizes the answers. In most cases, I link to the original comment thread, which you can read for details.

How is Oil different than bash or zsh?

Oil is taking shell seriously as a programming language, rather than treating it as a text-based UI that can be abused to write programs.

To see why this is valuable, consider these two groups of shell users:

1. People who use shell to type a few commands here and there.
2. People who write scripts, which may get into the hundreds or even thousands of lines.

Oil is aimed at group 2. If you're in group 1, there's admittedly no reason to use it right now.

However, group 2 writes scripts for group 1 to use! So I believe the benefits of Oil will eventually bubble up.

In other words, I'm building a solid foundation for a few more decades of shell usage.

• Original question and answer (reddit.com)
• Another comment with more detail

I don't understand. Why not use a different a programming language?

It's important to be compatible with existing code. You might not personally use shell as a programming language, but all Unix users still rely on big shell programs. It's often used at build time, but it's still used at runtime too, e.g. on embedded Linux devices.

Some of this code is old, but much of it is new. It's not a small amount of code, either. Examples:

• Kubernetes, a brand new cluster manager from Google (released around 2014), has 48,000+ lines of shell in its repo.
• Many distros use the bash-completion project for interactive completions, and it has 26,000 lines of shell.

Original question and answer (reddit.com)

Shouldn't we discourage people from writing shell scripts?

There are three problems with that:

1. It won't work. It would be like trying to convince people who are paid to write PHP not to write PHP. Many people have wasted breath on that, but important sites like Wikipedia are still written in hundreds of thousands of lines of PHP.
2. Even if a new line of shell never gets written, there will still be a huge installed base of shell scripts that you may need to understand (e.g. when they don't work).
3. Shell is still the best tool for many jobs. Most new "cloud" projects rely on Linux system images, in VMs or containers, and shell is the ideal language for creating such images. Even if you use a framework like Chef or Docker, you're still using bits of shell.

Shouldn't scripts over 100 lines be rewritten in Python or Ruby?

That's a reasonable choice in some circumstances. If everyone on your dev team knows Python, maintaining a shell script can be more costly than maintaining a Python script.

However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on.

I encountered a nice blog post, Replacing Shell Scripts with Python, which, in my opinion, inadvertently proves the opposite point. The Python version is longer and has more dependencies. In other words, it's more difficult to write and maintain.

Are you reinventing Perl?

It's true that Perl is closer to shell than Python and Ruby are. For example, the perl -pie idiom can take the place of awk and sed. However, Perl isn't an acceptable shell either:

• It doesn't have true pipelines (of processes). I believe you have to "shell out" for this.
• It doesn't have a concept of file descriptor state. How do you write the equivalent of my_shell_func 2> err.txt, where my_shell_func can invoke both functions and external commands?
• Perl has a function called grep(), but the real grep is better for many problems.

Also:

• Perl has been around for more than 30 years, and hasn't replaced shell. It hasn't replaced sed and awk either.
• Perl 6 and Python 3 are both less-suited to shell-like problems than their predecessors. In other words, we still need a new shell.

However, it's true that, in some respects, Oil is retreading the same ground as Perl. But Oil is more faithful to shell, and its syntax uses fewer punctuation characters. In other words, it's less like "line noise".

Threads:

• Perl vs Shell (reddit)
• Are you reinventing Perl? (Hacker News)

I'm still angry, and I don't want you to create a new shell.

You might be angry because you had to maintain a nasty shell script written by a coworker.

If that's the case, you should be helping Oil succeed! The only way to "kill bash" is to:

1. Reimplement it, then
2. Gradually migrate away from it.

This is analogous to how Facebook is moving away from PHP by developing a similar, but cleaner, language called Hack.

Perl, Python, and Ruby have all existed for over 25 years, but they haven't replaced shell. New shell scripts are being written every day.

• Angry Comment (Hacker News)
• My Response (Hacker News)

(Oil also has some similarity to CoffeeScript, which smoothed over some of JavaScript's rough edges and added syntactic sugar, but didn't stray from its core execution model. CoffeeScript was a success because it influenced subsequent versions of JavaScript.)

Should I limit myself to POSIX shell scripts?

I've seen this suggestion a lot, and there are entire books devoted to it. If your script is small, it may be a reasonable goal.

For bigger programs, limiting yourself to POSIX is not just inconvenient, it's also an ill-defined and virtually untestable concept. Evidence:

• The Debian project created a minimal shell variant called dash, which replaced bash. However, dash and the Debian Policy on Shell Scripts both allow non-POSIX features, like local. Local variables are essential for writing maintainable shell scripts, but POSIX doesn't mention them.

• Most shells parse assignment builtins differently than other builtins, but the POSIX shell grammar has no notion of an assignment. This issue surfaced as early as 2010, but as of 2018, it's not in a published spec.

  This issue isn't theoretical — bash and dash differ in practice, but POSIX doesn't specify which is correct.

  In other words, POSIX is incomplete and out of date. (However, I've discovered that shells are highly conformant with respect to things the standard does specify.)

As of 2018, I believe that OSH is a "better POSIX". POSIX is a descriptive specification and not a normative one. That means that it's an observation of how popular shells like ksh and bash happened to behave at a certain time. In other words, it's a compromise.

Similarly, OSH is based on extensive testing of the behavior of bash, dash, mksh, zsh, and busybox ash. That is, it uses the same philosophy as POSIX, but it specifies more of the language. Roughly speaking, spec tests are an executable specification.

More:

• Tilde Expansion: Another case where shells diverge in practice, but POSIX is silent.

How can you design a good language on top of shell?

It's indeed difficult, but after prototyping the OSH-to-Oil translator, I believe it can be done.

Does this mean that Oil isn't a good interactive shell?

I'm convinced that making shell a good programming language is a prerequisite for creating a good interactive shell.

If you've ever written a bash completion script from scratch, you might already agree. Shell doesn't have true functions, so bash's mechanism involves reading global variables likes $COMP_CWORD and mutating ones like $COMP_REPLY.

Although I'm a Vim user, I'm sometimes jealous that Emacs has a better programming language for customizing the UI.

Like Emacs, I expect that many of Oil's interactive features will be written in Oil, not C (or Python).

In summary, I'm prioritizing the language right now, but this will lay a solid foundation for interactive features. Also, I believe the OSH parser is a better foundation for completion than other shell parsers. Most shells don't appear to use their own parsers for interactive completion!

Why is it written in Python, and Why Python 2?

See the FAQ in Building Oil with the OPy Bytecode Compiler.

A Glimpse of Oil

Oil isn't yet implemented, and documenting it is a large project. But here is a glimpse:

Syntax

• Oil is statically parsed, which gives you more error messages.
• It uses a lossless syntax tree, which gives you better error messages.
• Oil will have two primary sublanguages, instead of four:
  • Shell has command, word, arith, and bool sublanguages.
  • Oil will have only command and expression sublanguages.
• Shell can be translated to Oil.

Semantics

• Oil has stricter runtime behavior, with set -o strict-array etc.
• Arrays are first-class data types in Oil; they won't be confused with strings.
• In addition to shell-like "functions" declared with the proc keyword, Oil will have Python/JavaScript-like functions declared with the func keyword. Funcs accept and return rich data types, which means they can:
  • Replace bizarre syntax for manipulating strings like ${s##--*}
  • Replace bizarre syntax for opening files like exec 5> config.log
  • Easily express quoting/escaping for any file format
• Oil should have support for structured data over pipes.
• Ruby-like blocks have many use cases in shell.

More Ambitious Ideas

• Combining shell, awk, and make.
  • the Awk paradigm of streaming "records".
  • the Make paradigm of incremental and parallel computation.
• An algebra of data frames, like R and tidyverse.

Many of these posts are tagged #oil-language. Also see #shell-the-good-parts.

Older Posts Explaining Oil

(1) Roadmap #5: Why OSH and Why Oil (October 2017). Linux distributions are heavy users of shell as a programming language, so OSH is initially focused on that use case.

(2) Project Goals (January 2017). Ilya Sher asked me why I created Oil, so I wrote this post and started a Wiki Page. I describe some more ambitious ideas, like creating a distributed shell. As of January 2018, I'm still struggling to replace bash, so those ideas are on the back burner!

Ilya also wrote a post explaining his shell project in FAQ form: Why Next Generation Shell?

I agree with all of his answers. Oil and NGS have largely the same motivations. The main difference between them is that OSH is compatible with POSIX shell and bash, but NGS isn't.

Conclusion

Leave a comment if there is something you don't understand, and I'll answer and possibly update the FAQ.

Appendix: More Ways to Understand Oil

Future blog posts can explain Oil in different ways:

OSH is to bash as LLVM/Clang is to GCC

OSH has a modular architecture:

• The lossless syntax tree can be used for running code, linting it, auto-formatting it, etc.
• The Oil interpreter can be separated from the libc kernel interface, for a Lua-like embeddable interpreter. The state and process modules are cleanly separated.

---

Thanks to Eric Higgins for feedback on a draft of this post.