How to share code between node.js apps?

I have several apps in node that all share a few modules that I've written. These modules are not available via npm. I would like to be able to share freely between apps, but I don't want to copy directories around, nor rely on Git to do so. And I'm not really big on using symlinks to do this either.

I would like to arrange directories something like this:

app1
 server.js
 node_modules
  (public modules from npm needed for app1)
 lib
  (my own modules specific to app1)

app2
 server.js
 node_modules
  (public modules from npm needed for app2)
 lib
  (my own modules specific to app2)

shared_lib
 (my own modules that are used in both app1 and app2)

The problem I'm seeing is that the modules in shared_lib seem to get confused as to where to find the modules that will be in the node_modules directory of whichever app they are running in. At least I think that is the problem.

So....what is a good way to do this that avoids having duplicates of files? (note that I don't care about duplicates of things in node_modules, since those aren't my code, I don't check them into Git, etc)

The npm documentation recommends using npm-link to create your own Node.js packages locally, and then making them available to other Node.js applications. It's a simple four-step process.

A typical procedure would be to first create a package with the following structure:

  hello
  | index.js
  | package.json

A typical implementation of these files would be:

index.js

  exports.world = function() {
     return('Hello World');
  }

package.json

  {
    "name": "hello",
    "version": "0.0.1",
    "private": true,
    "main": "index.js",
    "dependencies": {
    },
    "engines": {
    "node": "v0.6.x"
    }
  }

"private:true" ensures that npm will refuse to publish the package. This is a way to prevent accidental publication of private packages.

Next, navigate to the root of your Node.js package folder and run npm link to link the package globally so it can be used in other applications.

To use this package in another application, e.g., "hello-world", with the following directory structure:

 hello-world
 | app.js

Navigate to the hello-world folder and run:

 npm link hello

Now you can use it like any other npm package like so:

app.js

  var http = require('http');
  var hello = require('hello');

  var server = http.createServer(function(req, res) {
     res.writeHead(200);
     res.end(hello.world());
  });
  server.listen(8080);

I've got this working by having node_modules folders at different levels - node then automatically traverses upwards until it finds the module.

Note you don't have to publish to npm to have a module inside of node_modules - just use:

"private": true

Inside each of your private package.json files - for your project I would have the following:

app1
 server.js
 node_modules
  (public modules from npm needed for app1)
  (private modules locally needed for app1)
 
app2
 server.js
 node_modules
  (public modules from npm needed for app2)
  (private modules locally needed for app2)
 
node_modules
  (public modules from npm needed for app1 & app2)
  (private modules locally for app1 & app2)

The point is node.js has a mechanism for dealing with this already and it's awesome. Just combine it with the 'private not on NPM' trick and you are good to go.

In short a:

require('somemodule')

From app A or B would cascade upwards until it found the module - regardless if it lived lower down or higher up. Indeed - this lets you hot-swap the location without changing any of the require(...) statements.

node.js module documentation

Just use the correct path in your require call

For example in server.js that would be:

> var moduleName = require('../shared_lib/moduleName/module.js');

Its Important to know that as soon as your path is prefixed with '/', '../', or './' the path is relative to the calling file.

For further information about nodes module loading visit: http://nodejs.org/docs/latest/api/modules.html

Yes, you can reference shared_lib from app1, but then you run into a problem if you want to package and deploy app1 to some other environment, such as a web server on AWS.

In this case, you're better off installing your modules in shared_lib to app1 and app2 using "npm install shared_lib/module". It will also install all the dependencies of the shared_lib modules in app1 and app2 and deal with conflicts/duplicates.

See this: https://stackoverflow.com/questions/10386310/how-to-install-a-private-npm-module-without-my-own-registry

If you check out the node.js docs, you'll see that Node.js understands the package.json file format as well, at least cursorily.

Basically, if you have a directory named foo, and in that directory is a package.json file with the key-value pair: "main": "myCode.js", then if you try to require("foo") and it finds this directory with a package.json file inside, it will then use foo/myCode.js for the foo module.

So, with your directory structure, if each shared lib has it's own directory with such a simple package.json file inside, then your apps can get the shared libs by:

var lib1 = require('../shared_lib/lib1');
var lib2 = require('../shared_lib/lib2');

And that should work for both of these apps.

Another solution can be cloning files from the other places into this repo:

clone.js:

const path = require('path')
const fs = require('fs')

const shared = [
  {
    type: 'file',
    source: '../app1',
    files: [
      'src/file1',
      'src/file2',
      '...'
    ],
  },
]

function cloneFiles(source, files) {
  const Reset = '\x1b[0m'
  const FgGreen = '\x1b[32m'
  console.log(`---------- Cloning ${files.length} files from ${source} ----------`)

  for (const file of files) {
    const sourceFile = path.join(__dirname, '..', source, file)
    const targetFile = path.join(__dirname, '..', file)

    process.stdout.write(`📁 ${file} ... `)
    fs.copyFileSync(sourceFile, targetFile)
    console.log(`${FgGreen}Done!${Reset}`)
  }

  console.log(`---------- All done successfully ----------\n`)
}

;(() => {
  for (const item of shared) {
    switch (item.type) {
      case 'file':
        cloneFiles(item.source, item.files)
        break
    }
  }
})()

Then, in the package.json you can add this script and call it when you want to clone / sync files:

"clone": "node clone.js"