Asynchronous Control Flow Zen - The Ultimate Solution to "Promise Hell"

November 2, 2016

Most of us in the JavaScript development community have embraced Promises for asynchronous control flow, leaving callbacks and "Callback Hell" behind. But when working with Promise-heavy APIs, we often run into a different kind of pain - a "Promise Hell" if you will. But there is a solution – and it will bring you bliss!

Royanji Temple - https://www.flickr.com/photos/jimg944/3123212129

In the "old days" (which is defined as just a couple years ago in JavaScript time) we all used callbacks for asynchronous control flow. It was so ubiquitous and logical that most of us didn't even question the approach. That is until we started working more and more with asynchronous code.

Callback Hell

Every JavaScript developer that has worked with a lot of callback-based API calls has experienced it. The slow but sure creep to the right in your source code as you get deeper and deeper into a series of callbacks. They call in and call out, nest and intertwine until your control flow (and your mental model) are tied in knots.

Then came Promises. It was an awkward introduction, with some false starts (jQuery Deferred) and some initially slow implementations - but eventually some very fast libraries emerged, such as Bluebird and Zousan which removed speed as an impediment. Promises were adopted into the ES2015 spec and is now standard in all modern browsers. Now they are everywhere.

But for some of us who have used Promises extensively, we have found that while they definitely ease the burden of working with asynchronous code quite a bit, they too can get complex and convoluted. There are some great patterns of working with them in a series (then chains) or entirely in parallel (Promise.all), but in the real world, one is often faced with a complex mix of dependencies and control flow demands that force the use of temporary variables and nested thens. It can often result in a similar code creep – and sometimes even a kind of Promise Hell.

Ronald Chen describes it pretty well in this blog post: How To Escape Promise Hell. But even in his final "solution", things are far from simple and clean.

Let me explain the problem in a bit more detail, then our solution, and finally we will code up Ronald's example in a sublime, declarative manner that will be as clean and orderly as a Japanese rock garden!

The Problem

Most Promise libraries offer up a collection of "helper utilities" for managing a bunch of promises, such as Promise.all, Promise.race and Promise.map. These are very useful for specific cases where you want to process a bunch of Promises in a consistent way.

But with "real world" interaction with a variety of APIs, you often face dependencies and multi-argument functions that don't fit neatly into those models.

Lets take a look at the example Ronald Chen gives in the article mentioned above: You have 4 functions: connectDatabase(), findAllBooks(database), getCurrentUser(database), and pickTopRecommendation(books, user). These can't be all run in parallel with Promise.all since some functions depend on the return of previous functions. And that last function takes 2 arguments - both returned from previous promises.

So if we fall back to using standard Promise handling, we might write it like this:

return connectDatabase()
	.then(function(database) {
			return findAllBooks(database)
					.then(function(books) {
							return getCurrentUser(database)
									.then(function(user) {
											return pickTopRecommendation(books, user)
										})
						})
		})

See that creep to the right - descending into promise/then handlers.. it looks mighty familiar! It looks like Promise Hell!!

Yes, the ES2015 double arrow functions help a bit, but not much.

And worse than the code creep is the difficulty in understanding and reasoning about such code. We don't really need the findAllBooks function to resolve before calling getCurrentUser

As Ronald notes in his article, to make this workflow more optimized, we should call findAllBooks and getCurrentUser simultaneously - but only after calling connectDatabase. Then pickTopRecommendation must wait for all the others to complete:

const databasePromise = connectDatabase()

const booksPromise = databasePromise
	.then(findAllBooks)

const userPromise = databasePromise
	.then(getCurrentUser)

Promise.all([
		booksPromise,
		userPromise
	])
	.then(function(values) {
			const books = values[0]
			const user = values[1]
			return pickTopRecommentations(books, user)
		})

This works and is optimized, allowing for asynchronicity when possible and calling each function as soon as possible.

But this suffers in code clarity. Extra variables are needed to hold promises just to manage the workflow. More variables are needed to extract results from an array, in which positions in the Promise.all need to be matched up with the values array on the other end.

And you can imagine how these problems escalate as the problem increases in complexity. It becomes very prone to bugs and lack of optimization. And one change to an API can require a major reordering and restructuring of this control code.

And in Ronald's article this was proposed as the solution!

Control Flow Zen

Last month I blogged about an extension library (more of a utility belt than a library) called Zousan-Plus. It has a set of helper functions such as map, series, namedMap and tSeries. But the most recent addition is the most flexible, and most elegant way to handle these complex flow conditions. It results in code that is easy to understand, easy to reason about, and easy to change.  Best of all, it handles the optimization for you, so you always know you are maximizing your asynchronicity!

I call the function Zousan.eval - as it evaluates a set of functions (and/or other types of values - but generally they are functions) and builds an object with all the results. All you do is list the resources you wish to acquire and what their dependencies are. Order is not important:

return Zousan.eval(
		{ name: "database", value: connectDatabase },
		{ name: "books", value: findAllBooks, deps: [ "database" ] },
		{ name: "user", value: getCurrentUser, deps: [ "database "] },
		{ name: "top", value: pickTopRecommendation, deps: [ "books", "user" ] }
	).then(function(ob) { 
			return ob.top // we also have ob.user, ob.books and ob.database
		})

See how declarative that is? These items can be reordered and there is no effect - because each item is not evaluated until it has all the dependencies resolved. 

Furthermore, each function is called as soon as possible, maximizing your asynchronicity regardless of how complex or entangled the dependency tree is. Functions are called with the dependencies as arguments.

And notice the lack of temporary variables? We don't have to hold onto promises to be fed into later functions - Zousan.eval worries about that.

And value properties do not have to be functions that return promises - they can be previously defined promises, or functions that return other data types, or they can even simply be native values, such as a number:

Zousan.eval(
		{ name: "max", value: 100 },
		{ name: "items", value: getItems, deps: [ "max" ] },
		{ name: "disp", value: displayItems, deps: [ "items"] }
	)

As mentioned above, the item values in the deps array are used as arguments if the value specified on an item is a function. You can also simply specify a value in the deps array to pass into a function.

Zousan.eval is a single function and is remarkably small (42 lines) for the power that it affords. It has no dependencies on anything specific to Zousan - in fact, I went ahead and created a GitHub gist with Zousan.eval alone with no dependencies and attached to Promise.eval:

Loading Gist...

You can simply copy the code above and drop it into a project to experience Promise.eval goodness! Of course, if you want faster promises while supporting more browsers, you may want to use Zousan anyway.

What About ES2015 Generators, Async, Yield and Await?

These features are starting to appear in recent browser releases and are being used today by many developers via the babel transpiler. But I don't think they are particularly good at addressing the issues raised in this article. They may help with limiting callback hell, but they are focused more on hiding the asynchronous nature of code – attempting to make it look and feel like synchronous programming.

Kenny Kaye recently explored this topic and demonstrated the use of yield to achieve cleaner, simpler handling of async control flow. But lets take a look at his final solution to an asynchronous problem:

function setupNewUser(name) {
	return spawn(function*() {
		var invitations,
			newUser = yield createUser(name),
			friends = yield getFacebookFriends(name);

		if (friends) {
			invitations = yield inviteFacebookFriends(friends);
		}
	}
}

This is using Task.js, a 15k library that depends on generators - so the babel transpiler is recommended, which means tree shaking is recommended (webpack), etc. So it is a heavy-weight solution. Zousan, for comparison, is just over 2k - and if you don't need IE support or the added speed, you can leave that out.

But more importantly, notice in the above code that the getFacebookFriends() function does not depend on the completion of createUser(), yet it does not get called until createUser() returns. This is the problem with trying to ignore the asynchronous nature of these functions - you miss opportunities for optimization. This could result in much slower execution.

Lets try the above with Zousan.eval:

 

function setupNewUser(name) {
	return Zousan.eval(
		{ name: "newUser", value: createUser, deps: [name] },
		{ name: "friends", value: getFacebookFriends, deps: [name] },
		{ name: "invitations", value: inviteFacebookFriends, deps: [ "friends" ] }
		)
}

Using Zousan.eval, both createUser and getFacebookFriends will execute immediately and simultaneously, since their only dependency is an object passed in. Additionally, inviteFacebookFriends will execute as soon as friends is available resulting in potentially a much faster overall execution time than other techniques.

And since any errors that occur in resolving any of the promises will result in Zousan.eval rejecting, you can simply catch errors in the calling function, and do not need to qualify your calls with statements like "if (friends)" above.

So go forth and be asynchronous – be optimized – be declarative – and be zen!


If you or your organization could use a somewhat fanatical developer on your current or next project, lets talk. I am passionate about the user experience, easy to work with, and I get stuff done. Lets build something great!


Asynchronous Control Flow Zen - The Ultimate Solution to "Promise Hell" Tweet This!
Glenn is the founder of bluejava K.K., and has been providing consulting services and developing web/mobile applications for over 15 years. Glenn lives with his wife and two children in Fujisawa, Japan.
Comments
November 18, 2017
Great write-up Glenn. Sounds great that Zousan calls all the dependencies immediately. I think when I used Promise.all() to download a bunch of files, it was downloading them sequentially. I couldn't figure out how to make them run in parallel. Maybe Zousan could help if it tries to evaluate each promise as soon as possible. Hope I understood that correctly. Thanks for the article!
glenn
November 18, 2017
Thanks Michael! Actually, Promise.all kicks them all off immediately - what it doesn't do is allow any dependencies (or provide a very elegant means to access results of disparate functions - i.e. they are just result[0], result[1], ... Zousan.eval (now Zousan.evaluate) lets you assign name/value pairs with a complex dependency relationship, provides access to results of asynchronous dependent values to other functions, and finally returns the name/value for all evaluations, while optimizing their parallelism.
Minh Ha
July 22, 2019
I'd like to point out that caolan async.js has been able to do these complex sequences for ages, and people just ignore this fact when they mention "problems" with callback only to discover that they have to develop patterns to deal with them again in Promise or async/await. Doing this mixed sequence of parallel, sequential, nested mapSeries, nested mapParallel in caolan async is so simple. The given example can be written like this: async.auto({ connectDB: function(next) { GetDBConnection(next)//... Will call next (null, db); }, getBooks: ['connectDB', function(res, next) { findAllBooks(res['connectDB'], next); }], getUser: ['connectDB', function(res, next) { getCurrentUser(res['connectDB'], next); }], }, function(err, res) { if (err) { throw err; // Or call next level callback return; } const books = res['getBooks'] const user = values['getUser'] pickTopRecommentations(books, user) });
glenn
July 22, 2019
Oh wow, I didn't know about this function - thanks Minh! Thats a good point about software solutions being forgotten or overlooked and then people re-inventing them in a new context. I see that quite a bit. I will say, however, that the move to Promises is well worth it for many other reasons.
Minh Ha
July 23, 2019
Hi glenn, Yes, I agree with you that Promise, async/await, Rx are prevalent now for good reasons. Unfortunately, among them Promise seems to be the most unclear on how to do complex sequence so I agree with you that using something like Zousan Plus facilitates much better readability. IMHO, the Promise.all example is a BIG minus, i.e. having to rely positional arguments sucks, even destructuring does not help. Using something like Zousan Plus to collect results in an object should be in the introductory examples of Promise.
glenn
July 23, 2019
Cheers - I like that idea. :-) Please note that in this article I refer to Zouisan.eval - but it has been renamed to Zousan.evaluate to eliminate confusion with the eval keyword. I will edit this article soon. Thanks for your comments Minh!
(Comments currently disabled)