What's Happened Since February?

| blog | oilshell.org

2019-06-13

I haven't published a blog post in nearly four months, but the Oil project is still alive!

I've been busy coding, trying to push out a polished 0.6.0 release. I thought that would happen soon after I announced that OSH can run thousands of lines of interactive shell programs.

But it turned out that there were a lot of features, fixes, and polish necessary on top of that, and I was also derailed by a performance problem.

I'll review most of that work in this post.

I've been struggling with the fact that making a usable shell is a lot of work. One way to think about it is that bash is a 30-year-old program consisting of 140K lines of C code, and Oil has to replicate most of that functionality [1].

On the positive side, early users have filed high-quality bug reports, which have kept me busy. I need more help like this! Read on for details.

Seven Pre-Releases

Since announcing the 0.6.pre15 release in February, I've made 7 more pre-releases. Here I mention the general areas changed in each release, and link to the git log for those who want details.

0.6.pre16 on March 3rd

I worked on more aspects of the interactive shell, e.g.

• Shell autocompletion — both the interactive interface and the API for user plugins.
• Command history.
• Interactive signal handling (Ctrl-C, Ctrl-Z, SIGWINCH). Signals are tricky: I revisited them a few months after this release, and the work still isn't 100% done.

0.6.pre17 on March 21st

I statically typed the whole OSH front end with MyPy, which is around 10K lines of code. This work had four parts:

• Running unit tests with PyAnnotate, a program that records types at runtime.
• Editing and checking those annotations by hand. Without PyAnnotate, I would have had to write all annotations by hand.
• Writing a Zephyr ASDL -> MyPy converter. ASDL schemas already have types.
• Writing .pyi "header files" files for C extension modules.

Josh Nelson implemented:

• The builtin builtin.
• Additional $PS1 backslash codes.

0.6.pre18 on April 20th

As mentioned in February, the shell is too slow, and I'm eager to fix that problem.

During this release period, I prototyped "mycpp", a program that walks MyPy's typed AST and emits C++ code. This is one possible way to use types to speed up the OSH interpreter.

It can translate many of the programs in the mycpp/examples/ dir accurately, and there are significant speedups in most cases.

I haven't yet applied mycpp to Oil's source, but I plan to shortly.

Note: I mentioned OVM2 last fall, but consider it dormant for now. I intend to take the shortest path to speed up the shell, and leveraging MyPy and a C++ compiler should get us there sooner than trying to write my own VM.

• Also, Josh Nelson fixed a UI problem after Ctrl-C.

0.6.pre19 on May 9th

I made a pass over the whole codebase, checking that every error message is informative and has location information. These tests try to tickle every error:

• test/parse-errors
• test/runtime-errors

Good news: OSH objectively has the most precise error messages of any shell. For example:

# opt=typo
set -o $opt
       ^~~~
foo.sh:2: 'set' got invalid option 'typo'

I'll show more demos of this with the 0.6.0 release. I decided that the headline for that release will be OSH Has Useful Error Messages.

Bad news: After this release, I noticed a performance regression running Python's configure. After some debugging, it turns out that there was no regression. Instead, there had been a bug in the benchmark for months.

What changed? This release fixed the behavior of the $LINENO variable. Apparently, autoconf generates code to detect whether $LINENO works. When it didn't work in OSH, the configure script exec'd itself with /bin/sh, so it ran more quickly.

Fixing $LINENO means that OSH now runs the whole configure script, which reveals the slowness. :-( I hope that the mycpp work will fix this problem.

0.6.pre20 on May 14th

• Implemented integer ranges for brace expansion, e.g. {1..10..2}. As usual, shells disagree on the semantics of this feature, and I made OSH stricter and saner.
• Fixed set builtin behavior that autoconf scripts rely on.

0.6.pre21 on June 4th

• Implemented the printf builtin, along with printf -v (an odd syntax for assignment).
  • Unlike most printf implementations, I used a "proper" lexer and parser. This strategy has been invaluable for exposing bugs and handling corner cases.
• I integrated Python's pgen2 parser generator into the OSH parser. It will be the syntactic foundation for Oil's expression language, which will subsume and fix many messy shell constructs:
  • ${}
  • $(( )) and (( ))
  • [[ ]] and most of [ ]

More on that later.

0.6.pre22 on June 11th

• I made another pass over signals to fix important bugs reported by Crestwave and others. Essentially, I backported PEP 475 -- Retry system calls failing with EINTR to Python 2.
• Fix the semantics of "temp bindings" like PYTHONPATH=foo myfunc. Thanks to Michael Greenberg of the smoosh project for pointing this out. We learned that these bindings might be the feature where POSIX shells disagree most!
• As a followup, I tweaked the semantics of unset. This builtin interacts with shell's dynamic scope rule.

Contributors

Thanks again to the following people, mentioned above:

• Josh Nelson - for both patches and bug reports.
• Crestwave - for testing and bug reports. I still want to run the brainfuck interpreters in shell :-)
• Michael Greenberg - for testing and bug reports.
• okayzed - for testing, and pointing out a bug with Oil's file descriptor handling.
• Also, thanks to drwilly for significant work on find and xargs (not released yet).

Please continue to test OSH and file bugs!

Updated Docs and New Wiki Pages

Oil's documentation is currently sparse, but I'm not losing track of important points:

• osh-manual.md describes the oshrc startup file. OSH has a simpler behavior than other shells, and I expect this to be one of the first issues you have when starting to use it.
• known-differences.md catalogues more differences between OSH and other shells.

Designs on the Wiki:

• Projects Already Doing Something Like Shellac. It's common to scrape bash plugins to avoid rewriting completion logic. The barrier to shell-independent autocompletion seems to be parsing. See the Appendix of this post.
• Structured Data in Oil. This is future work. If you're interested, join the discussion on oilshell.zulipchat.com.

Developer tips:

• Using Zephyr ASDL: When I get questions from Oil developers, I often save the answers on the wiki.
• Signal Handling in Oil - Signal handling is a "global" concern and needs design documentation.

How to Help

I think Oil need more developers, but people are unlikely to become developers unless they're also users.

I've been suggesting that potential developers install OSH (which takes 30 seconds), and then:

• Report bugs that prevent you from using it. These are very useful, and have kept me busy for the last 4 months! Keep them coming.
• Try assembling an oshrc that you would use. My configuration is described on the wiki: How to Test OSH. I suspect that there will be friction, so let me know what happens on #oil-discuss on oilshell.zulipchat.com.
• Tell me a small feature what would motivate you use to it. There are many potential big features, but I'm looking for short term / small "carrots" to make OSH appealing.

If you're already working with the code, and have problems figuring it out, please ping me on Zulip.

The README.md and Contributing wiki page have helpful tips.

Bug Labels

I'm making consistent use of Github's issue tracker.

The #good-first-issue and #help-wanted labels generally up-to-date. Again, feel free to ping me on oilshell.zulipchat.com for ideas.

Other labels:

• #high-priority -- What I'm working on in the near future.
• #open-problem -- I don't know how to fix these bugs. I want you to help me!
• #affects-architecture -- The architecture of OSH has mostly settled, but there are a few medium-size changes I anticipate.
• I also use the #pending-release label for fixes that are completed but not released.

What's Next?

• Publish the 2019 edition of the FAQ.
  • Why is the Project Named "Oil"?
  • Why is it Written in Python?
  • Why Use It?
  • Why Not Use It?
  • What Happened to the Oil Language?
• Do an AMA on Reddit's /r/linux, scheduled for Monday the 17th. I will publicize this more later.
• Release OSH 0.6.0: A Shell With Useful and Precise Error Messages.
• Get feedback.
• Based on the feedback, figure out what goes into OSH 0.7.0. Hopefully this release will have features that motivate many people to use it.

Conclusion

I went over many topics quickly in this post. If you're interested in more detail, leave a comment or leave a message on #oil-discuss in Zulip.

Notes

[1] Wikipedia says that the first release of bash was on June 8th, 1989. So its 30th birthday was a few days ago. Happy Birthday bash!

Appendix: Blog Post Drafts

Although I plan to publish the FAQ first, I have drafts of the following blog posts:

• The Interactive Shell Needs a Principled Parser. I want to publish this post because its ideas have come up in multiple discussions, including this Hacker News thread.
• Dev Log #10: Experiments with Static Typing and Garbage Collection. I'm not sure if I will publish this, since it's more important to solve the speed problem than to document all the failed attempts to solve it. In short, I did a bunch of research and played with a few pieces of code:
  • Shed Skin: A translator from a subset of Python to C++. It performs global type inference, whereas mycpp uses explicit types. Somebody told me that they used Shed Skin in production, which suggested that mycpp was feasible.
  • hatlog: I forked a toy Prolog program to perform Python type inference. This was mainly a learning exercise.
  • oscar: This tiny garbage collector is fork()-friendly, like the Micro Python garbage collector. If I succeed in translating Oil to C++, I'll still have to solve the "deallocation problem".

If you have experience implementing garbage collection and Python-like data structures, I'd love to chat!