Why getting error mongod dead but subsys locked and Insufficient free space for journal files on Linux?

I have installed mongo-10gen mongo-10gen-server on Linux CentOS server.

I followed the steps from Link.

I have configured /etc/mongod.conf as -

logpath=/var/log/mongo/mongod.log
port=27017
dbpath=/var/lib/mongo

I have set port 27017 for mongo in iptables. To start mongo I used commands -

service mongod start and
mongo

It get started well, but after few days I am getting the error -

Tue Jan 29 08:41:54 [initandlisten] ERROR: Insufficient free space for journal files
Tue Jan 29 08:41:54 [initandlisten] Please make at least 3379MB available in /var/lib/mongo/journal or use --smallfiles
Tue Jan 29 08:41:54 [initandlisten]
Tue Jan 29 08:41:54 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
Tue Jan 29 08:41:54 dbexit:
Tue Jan 29 08:41:54 [initandlisten] shutdown: going to close listening sockets...
Tue Jan 29 08:41:54 [initandlisten] shutdown: going to flush diaglog...
Tue Jan 29 08:41:54 [initandlisten] shutdown: going to close sockets...
Tue Jan 29 08:41:54 [initandlisten] shutdown: waiting for fs preallocator...
Tue Jan 29 08:41:54 [initandlisten] shutdown: lock for final commit...
Tue Jan 29 08:41:54 [initandlisten] shutdown: final commit...
Tue Jan 29 08:41:54 [initandlisten] shutdown: closing all files...
Tue Jan 29 08:41:54 [initandlisten] closeAllFiles() finished
Tue Jan 29 08:41:54 [initandlisten] journalCleanup...
Tue Jan 29 08:41:54 [initandlisten] removeJournalFiles
Tue Jan 29 08:41:54 [initandlisten] shutdown: removing fs lock...
Tue Jan 29 08:41:54 dbexit: really exiting now

When I execute the command -

service mongod status

It gives Error -

mongod dead but subsys locked

Please help me to solve the problem of mongod dead but subsys locked and Insufficient free space for journals, terminating

You can add following to the config file provided when running mongod --config mongod.conf

For MongoDB 3.x (latest version)

storage:
   mmapv1:
      smallFiles: true

For version 2.6+

storage:
   smallFiles: true

For version 2.4 and less

smallfiles = true

Then just execute mongod to accept your config file (here it assumes that location of the config is /etc/mongodb.conf):

mongod -f /etc/mongodb.conf

Documentation for smallfiles parameter:

Set to true to modify MongoDB to use a smaller default data file size. 
Specifically, smallfiles reduces the initial size for data files and
limits them to 512 megabytes. The smallfiles setting also reduces the
size of each journal files from 1 gigabyte to 128 megabytes.

Start mongod instance using following command

mongod --dbpath /data/db --smallfiles

I was following the official guide at http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/

After using

$sudo service mongod start

I went to verify if MongoDB had started successfully by reviewing this log:

/var/log/mongodb/mongod.log

This is the problem that I found:

2014-11-11T12:54:05.808-0500 [initandlisten] ERROR: Insufficient free space for journal files
2014-11-11T12:54:05.808-0500 [initandlisten] Please make at least 3379MB available in /var/lib/mongodb/journal or use --smallfiles
2014-11-11T12:54:05.808-0500 [initandlisten]
2014-11-11T12:54:05.808-0500 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
2014-11-11T12:54:05.808-0500 [initandlisten] dbexit:
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: going to close listening sockets...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: going to flush diaglog...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: going to close sockets...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: waiting for fs preallocator...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: lock for final commit...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: final commit...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: closing all files...
2014-11-11T12:54:05.808-0500 [initandlisten] closeAllFiles() finished
2014-11-11T12:54:05.808-0500 [initandlisten] journalCleanup...
2014-11-11T12:54:05.808-0500 [initandlisten] removeJournalFiles
2014-11-11T12:54:05.814-0500 [initandlisten] shutdown: removing fs lock...
2014-11-11T12:54:05.814-0500 [initandlisten] dbexit: really exiting now

Solution:

At the end of the file /etc/mongod.conf I added this line:

smallfiles = true

After that, I restarted the mongod service:

$sudo service mongod restart

Then when I went to review the log, I realized that everything was perfect, and the problem was fixed:

2014-11-11T22:32:20.544-0500 ***** SERVER RESTARTED *****
2014-11-11T22:32:20.552-0500 [initandlisten] MongoDB starting : pid=5200 port=27017 dbpath=/var/lib/mongodb 64-bit host=jaimemontoya-VirtualBox
2014-11-11T22:32:20.552-0500 [initandlisten] db version v2.6.5
2014-11-11T22:32:20.552-0500 [initandlisten] git version: e99d4fcb4279c0279796f237aa92fe3b64560bf6
2014-11-11T22:32:20.552-0500 [initandlisten] build info: Linux build8.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-11-11T22:32:20.552-0500 [initandlisten] allocator: tcmalloc
2014-11-11T22:32:20.552-0500 [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/var/lib/mongodb", smallFiles: true }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2014-11-11T22:32:20.564-0500 [initandlisten] journal dir=/var/lib/mongodb/journal
2014-11-11T22:32:20.564-0500 [initandlisten] recover : no journal files present, no recovery needed
2014-11-11T22:32:20.738-0500 [initandlisten] waiting for connections on port 27017
2014-11-11T22:33:20.748-0500 [clientcursormon] mem (MB) res:36 virt:245
2014-11-11T22:33:20.748-0500 [clientcursormon]  mapped (incl journal view):64
2014-11-11T22:33:20.748-0500 [clientcursormon]  connections:0

For anyone having this issue despite having enough space for the journal file to be created.

My problem: I had more than enough space for the journal file to be created. Mongo gave me a connection failed issue. I used kill command to kill the mongod process which in turn gave me the subsys issue. Here is what worked for me:

Go to /var/lock/subsys and delete file mongod

now service mongod stopand then service mongod start

Started mongo shell now and everything seems to be running fine.

first

cd /usr/local/src/mongo/bin/<br>

then

./mongod --dbpath /usr/local/src/mongo/data/db/ --smallfiles

Just a note, which helped me. Though there were no technical issue related to mongo. It was just the system's space was insufficient. Clearing the system's cache worked out to me and mongo was back running smoothly.

yum clean all

Problem Background: I upgraded DB server from t2.micro to the larger instance without stopping database services. which in return created this lock file and throwing this error, mongod dead but subsys locked After this I was unable to use mongo shell.

If you have enough disk space. Then follow these steps,

1.sudo mongod (if you find below failed reason)

Failed to unlink socket file /tmp/mongodb-27017.sock Operation not permitted)

2. Go to /tmp/ directory

3. Remove mongodb-27017.sock file

4. Restart mongod with sudo service mongod restart