Bringing async/await to life in JavaScript

My dream has come true this week. I can now write clean asynchronous code in JavaScript: no callbacks, no intrusive control flow library, no ugly preprocessor. Just plain JavaScript!

This is made possible by a new feature of JavaScript called generator functions, which has been introduced by EcmaScript 6 and is now available in node.js (unstable version 0.11.2). I already blogged about generators a few times so I won’t get into the basics again here. The important thing is that ES6 introduces two small extensions to the language syntax:

• function*: the functions that you declare with a little twinkling star are generator functions. They execute in an unusual way and return generators.
• yield: this keyword lets you transfer control from a generator to the function that controls it.

And, even though these two language constructs were not orginally designed to have the async/await semantics found in other languages, it is possible to give them these semantics:

• The * in function* is your async keyword.
• yield is your await keyword.

Knowing this, you can write asynchronous code as if JavaScript had async/await keywords. Here is an example:

function* countLines(path) {
    var names = yield fs.readdir(path);
    var total = 0;
    for (var i = 0; i < names.length; i++) {
        var fullname = path + '/' + names[i];
        var count = (yield fs.readFile(fullname, 'utf8')).split('\n').length;
        console.log(fullname + ': ' + count);
        total += count;
    }
    return total;
}

function* projectLineCounts() {
    var total = 0;
    total += yield countLines(__dirname + '/../examples');
    total += yield countLines(__dirname + '/../lib');
    total += yield countLines(__dirname + '/../test');
    console.log('TOTAL: ' + total);
    return total;
}

Here, we have two asynchronous functions (countLines and projectLineCounts) that call each other and call node.js APIs (fs.readdir, fs.readFile). If you look carefully you’ll notice that these functions don’t call any special async helper API. Everything is done with our two markers: the little * marks declarations of asynchronous functions and yield marks calls to asynchronous functions. Just like async and await in other languages.

And it will work!

Galaxy

The magic comes from galaxy, a small library that I derived from my earlier work on streamline.js and generators.

Part of the magic is that the fs variable is not the usual node.js file system module; it is a small wrapper around that module:

var galaxy = require('galaxy');
var fs = galaxy.star(require('fs'));

The galaxy.star function converts usual callback-based node.js functions into generator functions that play well with the generator functions that we have written above.

The other part of the magic comes from the galaxy.unstar function which converts in the other direction, from generator functions to callback-based node.js functions. This unstar function allows us to transform projectLineCounts into a callback-based function that we can call as a regular node.js function:

var projectLineCountsCb = galaxy.unstar(projectLineCounts);

projectLineCountsCb(function(err, result) {
    if (err) throw err;
    console.log('CALLBACK RESULT: ' + result);
});

The complete example is available here.

The whole idea behind this API design is that galaxy lets you write code in two different spaces:

• The old callback space in which today’s node.js APIs live. In this space, you program with regular unstarred functions in continuation passing style (callbacks).
• The new generator space. In this space, you program in synchronous style with starred functions.

The star and unstar functions allow you to expose the APIs of one space into the other space. And that’s all you need to bring async/await to life in node.js.

Status

I assembled galaxy quickly from pieces that I had developed for streamline.js. So it needs a bit of polishing and the API may move a bit. Generator support in V8 and node.js is also brand new. So all of this is not yet ready for prime time but you can already play with it if you are curious.

I have introduced a galaxy.spin function to parallelize function calls. I’ll probably carry over some other goodies from the streamline project (funnel semaphore, asynchronous array functions, streams module, …).

I find it exciting that modules written in async/await style with galaxy don’t have any direct dependencies on the node.js callback convention. So, for example, it would be easy to write a browser variant of the star/unstar functions which would be aligned on the jQuery callback conventions, with separate callback and errback.

Also, another module was announced on the node.js mailing list this week: suspend. It takes the problem from a slightly different angle, by wrapping every generator function with a suspend call. It lets you consume node.js APIs directly and write functions that follow node’s callback pattern. This is an attractive option for library developers who want to stay close to node’s callback model. Take a look at the source; it’s really clever: only 16 locs! Galaxy is different in that it moves you to a different space where you can program in sync style with no additional API, just language keywords. Probably a more attractive option if you are writing applications because you’ll get leaner code if most of your calls are to your own APIs rather than to node’s APIs.

Happy */yield coding!