Node.js - asynchronous module loading

Is it possible to load a Node.js module asynchronously?

This is the standard code:

var foo = require("./foo.js"); // waiting for I/O
foo.bar();

But I would like to write something like this:

require("./foo.js", function(foo) {
    foo.bar();
});
// doing something else while the hard drive is crunching...

Is there a way how to do this? Or is there a good reason why callbacks in require aren't supported?

While require is synchronous, and Node.js does not provide an asynchronous variant out of the box, you can easily build one for yourself.

First of all, you need to create a module. In my example I am going to write a module that loads data asynchronously from the file system, but of course YMMV. So, first of all the old-fashioned, not wanted, synchronous approach:

var fs = require('fs');
var passwords = fs.readFileSync('/etc/passwd');

module.exports = passwords;

You can use this module as usual:

var passwords = require('./passwords');

Now, what you want to do is turn this into an asynchronous module. As you can not delay module.exports, what you do instead is instantly export a function that does the work asynchronously and calls you back once it is done. So you transform your module into:

var fs = require('fs');
module.exports = function (callback) {
  fs.readFile('/etc/passwd', function (err, data) {
    callback(err, data);
  });
};

Of course you can shorten this by directly providing the callback variable to the readFile call, but I wanted to make it explicit here for demonstration purposes.

Now when you require this module, at first, nothing happens, as you only get a reference to the asynchronous (anonymous) function. What you need to do is call it right away and provide another function as callback:

require('./passwords')(function (err, passwords) {
  // This code runs once the passwords have been loaded.
});

Using this approach you can, of course, turn any arbitrary synchronous module initialization to an asynchronous one. But the trick is always the same: Export a function, call it right from the require call and provide a callback that continues execution once the asynchronous code has been run.

Please note that for some people

require('...')(function () { ... });

may look confusing. Hence it may be better (although this depends on your actual scenario) to export an object with an asynchronous initialize function or something like that:

var fs = require('fs');
module.exports = {
  initialize: function (callback) {
    fs.readFile('/etc/passwd', function (err, data) {
      callback(err, data);
    });
  }
};

You can then use this module by using

require('./passwords').initialize(function (err, passwords) {
  // ...
});

which may be slightly better readable.

Of course you can also use promises or any other asynchronous mechanism which makes your syntax look nicer, but in the end, it (internally) always comes down to the pattern I just described here. Basically, promises & co. are nothing but syntactic sugar over callbacks.

Once you build your modules like this, you can even build a requireAsync function that works like you initially suggested in your question. All you have to do is stick with a name for the initialization function, such as initialize. Then you can do:

var requireAsync = function (module, callback) {
  require(module).initialize(callback);
};

requireAsync('./passwords', function (err, passwords) {
  // ...
});

Please note, that, of course, loading the module will still be synchronous due to the limitations of the require function, but all the rest will be asynchronous as you wish.

One final note: If you want to actually make loading modules asynchronous, you could implement a function that uses fs.readFile to asynchronously load a file, and then run it through an eval call to actually execute the module, but I'd highly recommend against this: One the one hand, you lose all the convenience features of request such as caching & co., on the other hand you'll have to deal with eval - and as we all know, eval is evil. So don't do it.

Nevertheless, if you still want to do it, basically it works like this:

var requireAsync = function (module, callback) {
  fs.readFile(module, { encoding: 'utf8' }, function (err, data) {
    var module = {
      exports: {}
    };
    var code = '(function (module) {' + data + '})(module)';
    eval(code);
    callback(null, module);
  });
};

Please note that this code is not "nice", and that it lacks any error handling, and any other capabilities of the original require function, but basically, it fulfills your demand of being able to asynchronously load synchronously designed modules.

Anyway, you can use this function with a module like

module.exports = 'foo';

and load it using:

requireAsync('./foo.js', function (err, module) {
  console.log(module.exports); // => 'foo'
});

Of course you can export anything else as well. Maybe, to be compatible with the original require function, it may be better to run

callback(null, module.exports);

as last line of your requireAsync function, as then you have direct access to the exports object (which is the string foo in this case). Due to the fact that you wrap the loaded code inside of an immediately executed function, everything in this module stays private, and the only interface to the outer world is the module object you pass in.

Of course one can argue that this usage of evil is not the best idea in the world, as it opens up security holes and so on - but if you require a module, you basically do nothing else, anyway, than eval-uating it. The point is: If you don't trust the code, eval is the same bad idea as require. Hence in this special case, it might be fine.

If you are using strict mode, eval is no good for you, and you need to go with the vm module and use its runInNewContext function. Then, the solution looks like:

var requireAsync = function (module, callback) {
  fs.readFile(module, { encoding: 'utf8' }, function (err, data) {
    var sandbox = {
      module: {
        exports: {}
      }
    };
    var code = '(function (module) {' + data + '})(module)';
    vm.runInNewContext(code, sandbox);
    callback(null, sandbox.module.exports); // or sandbox.module…
  });
};

Hope this helps.

The npm module async-require can help you to do this.

Install

npm install --save async-require

Usage

var asyncRequire = require('async-require');
    
// Load script myModule.js 
asyncRequire('myModule').then(function (module) {
    // module has been exported and can be used here
    // ...
});

The module uses vm.runInNewContext(), a technique discussed in the accepted answer. It has bluebird as a dependency.

(This solution appeared in an earlier answer but that was deleted by review.)

Yes - export function accepting callback or maybe even export full featured promise object.

// foo.js + callback:
module.exports = function(cb) {
   setTimeout(function() {
      console.log('module loaded!');
      var fooAsyncImpl = {};
      // add methods, for example from db lookup results
      fooAsyncImpl.bar = console.log.bind(console);
      cb(null, fooAsyncImpl);
   }, 1000);
}

// usage
require("./foo.js")(function(foo) {
    foo.bar();
});

// foo.js + promise
var Promise = require('bluebird');
module.exports = new Promise(function(resolve, reject) {
   // async code here;
});

// using foo + promises
require("./foo.js").then(function(foo) {
    foo.bar();
});

Andrey's code below is the simplest answer which works, but his had a small mistake so I'm posting the correction here as answer. Also, I'm just using callbacks, not bluebird / promises like Andrey's code.

/* 1. Create a module that does the async operation - request etc */

// foo.js + callback:
module.exports = function(cb) {
   setTimeout(function() {
      console.log('module loaded!');
      var foo = {};
      // add methods, for example from db lookup results
      foo.bar = function(test){
          console.log('foo.bar() executed with ' + test);
      };
      cb(null, foo);

   }, 1000);
}

/* 2. From another module you can require the first module and specify your callback function */

// usage
require("./foo.js")(function(err, foo) {
    foo.bar('It Works!');
});

/* 3. You can also pass in arguments from the invoking function that can be utilised by the module - e.g the "It Works!" argument */