Asynchronous Control Flow Zen - The Ultimate Solution to "Promise Hell"

In the "old days" (which is defined as just a couple years ago in JavaScript time) we all used callbacks for asynchronous control flow. It was so ubiquitous and logical that most of us didn't even question the approach. That is until we started working more and more with asynchronous code.

Callback Hell

Every JavaScript developer that has worked with a lot of callback-based API calls has experienced it. The slow but sure creep to the right in your source code as you get deeper and deeper into a series of callbacks. They call in and call out, nest and intertwine until your control flow (and your mental model) are tied in knots.

Then came Promises. It was an awkward introduction, with some false starts (jQuery Deferred) and some initially slow implementations - but eventually some very fast libraries emerged, such as Bluebird and Zousan which removed speed as an impediment. Promises were adopted into the ES2015 spec and is now standard in all modern browsers. Now they are everywhere.

But for some of us who have used Promises extensively, we have found that while they definitely ease the burden of working with asynchronous code quite a bit, they too can get complex and convoluted. There are some great patterns of working with them in a series (then chains) or entirely in parallel (Promise.all), but in the real world, one is often faced with a complex mix of dependencies and control flow demands that force the use of temporary variables and nested thens. It can often result in a similar code creep – and sometimes even a kind of Promise Hell.

Ronald Chen describes it pretty well in this blog post: How To Escape Promise Hell. But even in his final "solution", things are far from simple and clean.

Let me explain the problem in a bit more detail, then our solution, and finally we will code up Ronald's example in a sublime, declarative manner that will be as clean and orderly as a Japanese rock garden!

The Problem

Most Promise libraries offer up a collection of "helper utilities" for managing a bunch of promises, such as Promise.all, Promise.race and Promise.map. These are very useful for specific cases where you want to process a bunch of Promises in a consistent way.

But with "real world" interaction with a variety of APIs, you often face dependencies and multi-argument functions that don't fit neatly into those models.

Lets take a look at the example Ronald Chen gives in the article mentioned above: You have 4 functions: connectDatabase(), findAllBooks(database), getCurrentUser(database), and pickTopRecommendation(books, user). These can't be all run in parallel with Promise.all since some functions depend on the return of previous functions. And that last function takes 2 arguments - both returned from previous promises.

So if we fall back to using standard Promise handling, we might write it like this:

return connectDatabase()
	.then(function(database) {
			return findAllBooks(database)
					.then(function(books) {
							return getCurrentUser(database)
									.then(function(user) {
											return pickTopRecommendation(books, user)
										})
						})
		})

See that creep to the right - descending into promise/then handlers.. it looks mighty familiar! It looks like Promise Hell!!

Yes, the ES2015 double arrow functions help a bit, but not much.

And worse than the code creep is the difficulty in understanding and reasoning about such code. We don't really need the findAllBooks function to resolve before calling getCurrentUser. 

As Ronald notes in his article, to make this workflow more optimized, we should call findAllBooks and getCurrentUser simultaneously - but only after calling connectDatabase. Then pickTopRecommendation must wait for all the others to complete:

const databasePromise = connectDatabase()

const booksPromise = databasePromise
	.then(findAllBooks)

const userPromise = databasePromise
	.then(getCurrentUser)

Promise.all([
		booksPromise,
		userPromise
	])
	.then(function(values) {
			const books = values[0]
			const user = values[1]
			return pickTopRecommentations(books, user)
		})

This works and is optimized, allowing for asynchronicity when possible and calling each function as soon as possible.

But this suffers in code clarity. Extra variables are needed to hold promises just to manage the workflow. More variables are needed to extract results from an array, in which positions in the Promise.all need to be matched up with the values array on the other end.

And you can imagine how these problems escalate as the problem increases in complexity. It becomes very prone to bugs and lack of optimization. And one change to an API can require a major reordering and restructuring of this control code.

And in Ronald's article this was proposed as the solution!

Control Flow Zen

Last month I blogged about an extension library (more of a utility belt than a library) called Zousan-Plus. It has a set of helper functions such as map, series, namedMap and tSeries. But the most recent addition is the most flexible, and most elegant way to handle these complex flow conditions. It results in code that is easy to understand, easy to reason about, and easy to change.  Best of all, it handles the optimization for you, so you always know you are maximizing your asynchronicity!

I call the function Zousan.eval - as it evaluates a set of functions (and/or other types of values - but generally they are functions) and builds an object with all the results. All you do is list the resources you wish to acquire and what their dependencies are. Order is not important:

return Zousan.eval(
		{ name: "database", value: connectDatabase },
		{ name: "books", value: findAllBooks, deps: [ "database" ] },
		{ name: "user", value: getCurrentUser, deps: [ "database "] },
		{ name: "top", value: pickTopRecommendation, deps: [ "books", "user" ] }
	).then(function(ob) { 
			return ob.top // we also have ob.user, ob.books and ob.database
		})

See how declarative that is? These items can be reordered and there is no effect - because each item is not evaluated until it has all the dependencies resolved. 

Furthermore, each function is called as soon as possible, maximizing your asynchronicity regardless of how complex or entangled the dependency tree is. Functions are called with the dependencies as arguments.

And notice the lack of temporary variables? We don't have to hold onto promises to be fed into later functions - Zousan.eval worries about that.

And value properties do not have to be functions that return promises - they can be previously defined promises, or functions that return other data types, or they can even simply be native values, such as a number:

Zousan.eval(
		{ name: "max", value: 100 },
		{ name: "items", value: getItems, deps: [ "max" ] },
		{ name: "disp", value: displayItems, deps: [ "items"] }
	)

As mentioned above, the item values in the deps array are used as arguments if the value specified on an item is a function. You can also simply specify a value in the deps array to pass into a function.

Zousan.eval is a single function and is remarkably small (42 lines) for the power that it affords. It has no dependencies on anything specific to Zousan - in fact, I went ahead and created a GitHub gist with Zousan.eval alone with no dependencies and attached to Promise.eval:

You can simply copy the code above and drop it into a project to experience Promise.eval goodness! Of course, if you want faster promises while supporting more browsers, you may want to use Zousan anyway.

What About ES2015 Generators, Async, Yield and Await?

These features are starting to appear in recent browser releases and are being used today by many developers via the babel transpiler. But I don't think they are particularly good at addressing the issues raised in this article. They may help with limiting callback hell, but they are focused more on hiding the asynchronous nature of code – attempting to make it look and feel like synchronous programming.

Kenny Kaye recently explored this topic and demonstrated the use of yield to achieve cleaner, simpler handling of async control flow. But lets take a look at his final solution to an asynchronous problem:

function setupNewUser(name) {
	return spawn(function*() {
		var invitations,
			newUser = yield createUser(name),
			friends = yield getFacebookFriends(name);

		if (friends) {
			invitations = yield inviteFacebookFriends(friends);
		}
	}
}

This is using Task.js, a 15k library that depends on generators - so the babel transpiler is recommended, which means tree shaking is recommended (webpack), etc. So it is a heavy-weight solution. Zousan, for comparison, is just over 2k - and if you don't need IE support or the added speed, you can leave that out.

But more importantly, notice in the above code that the getFacebookFriends() function does not depend on the completion of createUser(), yet it does not get called until createUser() returns. This is the problem with trying to ignore the asynchronous nature of these functions - you miss opportunities for optimization. This could result in much slower execution.

Lets try the above with Zousan.eval:

function setupNewUser(name) {
	return Zousan.eval(
		{ name: "newUser", value: createUser, deps: [name] },
		{ name: "friends", value: getFacebookFriends, deps: [name] },
		{ name: "invitations", value: inviteFacebookFriends, deps: [ "friends" ] }
		)
}

Using Zousan.eval, both createUser and getFacebookFriends will execute immediately and simultaneously, since their only dependency is an object passed in. Additionally, inviteFacebookFriends will execute as soon as friends is available resulting in potentially a much faster overall execution time than other techniques.

And since any errors that occur in resolving any of the promises will result in Zousan.eval rejecting, you can simply catch errors in the calling function, and do not need to qualify your calls with statements like "if (friends)" above.

So go forth and be asynchronous – be optimized – be declarative – and be zen!