Writing Maintainable Event-Driven Code

I have just recently started playing with event-driven architectures, coming from a pretty standard object-oriented mindset.

The first thing I noticed was that the difficulty in understanding and tracing through programs seems to increase exponentially with the size of the program. While small pet projects are easy to follow, it feels like the code will rapidly turn to spaghetti.

I understand that I am new to this development mindset and not all of my object oriented worries carry over. Are there any resources on writing maintainable, understandable event-driven code? What do people who use node.js or Twisted or Event Machine do about this?

I did a talk on this topic at Yahoo last year.

Try to look at these articles:

• Understanding the node.js event loop
• Control Flow in Node
• Control Flow in Node Part II

Martyn Loughran wrote an excellent short article entirely about avoiding the callback spaghetti.

What I really enjoyed about his article is the process of improving spaghetti into something nice and clean; it might seem a little formalized at first, but when you see the end result I think you'll agree he shows real artistry in clean, legible, code.

I'll use Python as an example as that is what I am using to build huge distributed applications right now.

Twisted python allows for a very imperative style using either inlinecallbacks or (slightly uglier) deferredGenerator styles. These methods allow you to write procedures that use event driven callback code that is much easier to read and understand. The implementation turns your function into a lazy sequence that yields a sequence of deferreds.

Specifically, you don't have to build a deeply nested set of callback functions/lambdas/closures, and can instead yield control of a function back to the event loop at arbitrary points. You can mentally re-label this as coroutines or cooperative multitasking if you like. It gets the job done. An example would be (using the uglier deferredGenerator style) like this:

@defer.deferredGenerator
def foo(arg):
    bar = nonBlockingFunction(foo)
    baz = waitForDeferred(aFunctionThatReturnsADeferredToo(bar))
    yield baz #Returns control to the event loop
    output = baz.getResult() #This gets the output of aFunctionThat...Too above
    yield output #This is how we return a result instead of using return

@defer.deferredGenerator
def aFunctionThatReturnsADeferredToo(put_bar_here):
    """Stuff happens here...."""
    ...etc...

There is another post here that shows the inlineCallbacks method, which is cleaner, but requires python 2.5 or newer (meaning not under Centos/RHEL 5 series, which I am sadly stuck with for my app). If you can use it DO SO.

As you can see, this looks like the old school python imperative stuff you know and love, but is WAY easier to maintain without a ton of nested functions and lambdas. I still wish python had blocks though.

As for debugging, you can turn on twisted reactor debugging using the defer.setDebugging(True) call somewhere in your initialization code. This will attach the original traceback that raised an exception in your code, so that you can trivially see where the error ACTUALLY occurred. Just remember to redact the setDebugging statement before going production, because it results in a HUGE amount of extra introspection (watch it in strace if you want to be utterly horrified).

For Twisted, instead of using the old deferredGenerator I recommend the inlineCallbacks; It allows you to fully write blocking-style code and still play nicely with the event loop.

@defer.inlineCallbacks
def foo(arg):
    bar = nonBlockingFunction(foo)
    output = yield FunctionThatReturnsADeferredToo(bar)
    defer.returnValue(output) #This is how we return a result instead of using return

Obviously there are already best practices and models that will continue to develop over time.

However, consider also the possibility that evented programming provides the opportunity for "small pet projects" to interact with each other. Imagine a world where thousands of distributed individual projects interacted in real time through user-defined callbacks.

Users and developers would be able to rewire the web and applications over existing protocols from the top down instead of relying on existing application design. Application designers would then be free to focus on individual use cases instead of providing one-size-fits-all solutions or worrying about every possible contingency.

Check out Web Hooks and look at how services like Twilio are already operating

My only advice is think functional.