Is it safe to delete the journal file of mongodb?

If I delete the 3.1G journal file, sudo service mongodb restart will fail. However, this file is taking too much space. How can I solve this problem? How can I remove it?

bash$ du -sh /var/lib/mongodb/*
4.0K	_tmp
65M	auction_development.0
128M	auction_development.1
17M	auction_development.ns
3.1G	journal
4.0K	mongod.lock

TL;DR: You have two options. Use the --smallfiles startup option when starting MongoDB to limit the size of the journal files to 128MB, or turn off journalling using the --nojournal option. Using --nojournal in production is usually a bad idea, and it often makes sense to use different write concerns also in development so you don't have different code in dev and prod.

The long answer: No, deleting the journal file isn't safe. The idea of journalling is this:

A write comes in. Now, to make that write persistent (and the database durable), the write must somehow go to the disk.

Unfortunately, writes to the disk take eons compared to writes to the RAM, so the database is in a dilemma: not writing to the disk is risky, because an unexpected shutdown would cause data loss. But writing to the disk for every single write operation will decrease the database's performance so badly that it becomes unusable for practical purposes.

Now instead of writing to the data files themselves, and instead of doing it for every request, the database will simply append to a journal file where it stores all the operations that haven't been committed to the actual data files yet. This is a lot faster, because the file is already 'hot' since it's read and written to all the time, and it's only one file, not a bunch of files, and lastly, because it writes all pending operations in a batch every 100ms by default. Deleting this file in the middle of something wreaks havoc.

As explained in mnemosyn's answer, journaling is essential to the storage engines. Luckily, it can be controlled to some extent. The following was written for the MMAPv1 storage engine, which was the default until MongoDB 3.2. Then, WiredTiger became the engine of choice, to which more information can be found at the bottom of this answer.

MMAPv1

MongoDB < 2.6 (Non-YAML Config)

For our development server, we used the following procedure:

cp -p /etc/mongodb.conf /etc/mongodb.conf.orig
vi /etc/mongodb.conf

Now, insert

smallfiles=true

into the mongodb.conf, then save. smallfiles limits the journal file to 128MB.

service mongodb stop
rm -rf /var/lib/mongodb/journal/*
service mongodb start

MongoDB >= 2.6 (YAML Config)

If you're using MMAPv1 with the YAML config style, use the same step to backup the config as above, but into the

  mmapv1:

config block, insert

    smallFiles: true 

. Afterwards, proceed as above, restarting the server whilst removing the journals.

WiredTiger (MongoDB >=3.0, default since 3.2)

On development machines, journal files under WiredTiger should be somewhat smaller by default than under MMAPv1, as journal compression is enabled by default. According to the documentation, "WiredTiger journal files for MongoDB have a maximum size limit of approximately 100 MB". It will "create checkpoints (i.e. write the snapshot data to disk) at intervals of 60 seconds or 2 gigabytes of journal data."

Thus, if you're only running a low amount of requests (with little data to change) on your database, the journal files using WiredTiger should not exceed a low multiple of 100 MB. The size of the journal files seems not to be configurable, however.

mongodb has evolved since. Now its v3.4.1 stable.
I am on v3.2 here's how:
uncomment # mmapv1: so it looks like:

  mmapv1:
    smallFiles: true 

if you have a different version look for storage Options on the reference/configuration-options page.

don't forget to empty the journal

sudo service mongodb stop
sudo rm -rf /var/lib/mongodb/journal/*
sudo service mongodb start