Neo4j WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the Neo4j manual

Linux Problem Overview

I installed Neo4j on Ubuntu 12.04 using these instructions: http://www.neo4j.org/download/linux

wget -O - http://debian.neo4j.org/neotechnology.gpg.key | apt-key add - 
echo 'deb http://debian.neo4j.org/repo stable/' > /etc/apt/sources.list.d/neo4j.list
apt-get update
apt-get install neo4j

I made sure to have the right jdk:

root@precise64:~# java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

And when I startup neo4j I get this error:

> WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. > See the Neo4j manual.

So I follow the instructions here: http://docs.neo4j.org/chunked/1.6.2/configuration-linux-notes.html

And add these contents to /etc/security/limits.conf

neo4j   soft    nofile  40000
neo4j   hard    nofile  40000

And uncomment this line in /etc/pam.d/su

session    required   pam_limits.so

After I restart the server I validate the new limit

neo4j@precise64:~$ ulimit -n
40000

Then I restart my server. I still get the same error when starting up.

root@precise64:~# service neo4j-service start
WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the Neo4j manual.
Using additional JVM arguments:  -server -XX:+DisableExplicitGC -Dorg.neo4j.server.properties=conf/neo4j-server.properties -Djava.util.logging.config.file=conf/logging.properties -Dlog4j.configuration=file:conf/log4j.properties -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
Starting Neo4j Server...WARNING: not changing user
process [3161]... waiting for server to be ready....... OK.
http://localhost:7474/ is ready.

Even though the service is running as the neo4j user:

root@precise64:~# ps -ef | grep 3161
neo4j     3161     1  2 17:43 pts/0    00:00:10 /usr/bin/java -cp /var/lib/neo4j/lib/concurrentlinkedhashmap-lru-1.3.1.jar:/var/lib/neo4j/lib/geronimo-jta_1.1_spec-1.1.1.jar:/var/lib/neo4j/lib/lucene-core-3.6.2.jar:/var/lib/neo4j/lib/neo4j-cypher-2.0.0.jar:/var/lib/neo4j/lib/neo4j-cypher-commons-2.0.0.jar:/var/lib/neo4j/lib/neo4j-cypher-compiler-1.9-2.0.0.jar:/var/lib/neo4j/lib/neo4j-cypher-compiler-2.0-2.0.0.jar:/var/lib/neo4j/lib/neo4j-graph-algo-2.0.0.jar:/var/lib/neo4j/lib/neo4j-graph-matching-2.0.0.jar:/var/lib/neo4j/lib/neo4j-jmx-2.0.0.jar:/var/lib/neo4j/lib/neo4j-kernel-2.0.0.jar:/var/lib/neo4j/lib/neo4j-lucene-index-2.0.0.jar:/var/lib/neo4j/lib/neo4j-shell-2.0.0.jar:/var/lib/neo4j/lib/neo4j-udc-2.0.0.jar:/var/lib/neo4j/lib/org.apache.servicemix.bundles.jline-0.9.94_1.jar:/var/lib/neo4j/lib/parboiled-core-1.1.6.jar:/var/lib/neo4j/lib/parboiled-scala_2.10-1.1.6.jar:/var/lib/neo4j/lib/scala-library-2.10.3.jar:/var/lib/neo4j/lib/server-api-2.0.0.jar:/var/lib/neo4j/system/lib/asm-3.1.jar:/var/lib/neo4j/system/lib/bcprov-jdk16-140.jar:/var/lib/neo4j/system/lib/commons-beanutils-1.8.0.jar:/var/lib/neo4j/system/lib/commons-beanutils-core-1.8.0.jar:/var/lib/neo4j/system/lib/commons-collections-3.2.1.jar:/var/lib/neo4j/system/lib/commons-compiler-2.6.1.jar:/var/lib/neo4j/system/lib/commons-configuration-1.6.jar:/var/lib/neo4j/system/lib/commons-digester-1.8.1.jar:/var/lib/neo4j/system/lib/commons-io-1.4.jar:/var/lib/neo4j/system/lib/commons-lang-2.4.jar:/var/lib/neo4j/system/lib/commons-logging-1.1.1.jar:/var/lib/neo4j/system/lib/jackson-core-asl-1.9.7.jar:/var/lib/neo4j/system/lib/jackson-jaxrs-1.9.7.jar:/var/lib/neo4j/system/lib/jackson-mapper-asl-1.9.7.jar:/var/lib/neo4j/system/lib/janino-2.6.1.jar:/var/lib/neo4j/system/lib/javax.servlet-3.0.0.v201112011016.jar:/var/lib/neo4j/system/lib/jcl-over-slf4j-1.6.1.jar:/var/lib/neo4j/system/lib/jersey-core-1.9.jar:/var/lib/neo4j/system/lib/jersey-multipart-1.9.jar:/var/lib/neo4j/system/lib/jersey-server-1.9.jar:/var/lib/neo4j/system/lib/jetty-http-9.0.5.v20130815.jar:/var/lib/neo4j/system/lib/jetty-io-9.0.5.v20130815.jar:/var/lib/neo4j/system/lib/jetty-security-9.0.5.v20130815.jar:/var/lib/neo4j/system/lib/jetty-server-9.0.5.v20130815.jar:/var/lib/neo4j/system/lib/jetty-servlet-9.0.5.v20130815.jar:/var/lib/neo4j/system/lib/jetty-util-9.0.5.v20130815.jar:/var/lib/neo4j/system/lib/jetty-webapp-9.0.5.v20130815.jar:/var/lib/neo4j/system/lib/jetty-xml-9.0.5.v20130815.jar:/var/lib/neo4j/system/lib/jsr311-api-1.1.2.r612.jar:/var/lib/neo4j/system/lib/logback-access-1.0.9.jar:/var/lib/neo4j/system/lib/logback-classic-1.0.9.jar:/var/lib/neo4j/system/lib/logback-core-1.0.9.jar:/var/lib/neo4j/system/lib/mimepull-1.6.jar:/var/lib/neo4j/system/lib/neo4j-browser-2.0.0.jar:/var/lib/neo4j/system/lib/neo4j-server-2.0.0-static-web.jar:/var/lib/neo4j/system/lib/neo4j-server-2.0.0.jar:/var/lib/neo4j/system/lib/rhino-1.7R3.jar:/var/lib/neo4j/system/lib/rrd4j-2.0.7.jar:/var/lib/neo4j/system/lib/slf4j-api-1.6.2.jar:/var/lib/neo4j/conf/ -server -XX:+DisableExplicitGC -Dorg.neo4j.server.properties=conf/neo4j-server.properties -Djava.util.logging.config.file=conf/logging.properties -Dlog4j.configuration=file:conf/log4j.properties -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dneo4j.home=/var/lib/neo4j -Dneo4j.instance=/var/lib/neo4j -Dfile.encoding=UTF-8 org.neo4j.server.Bootstrapper
root      3238  2056  0 17:49 pts/0    00:00:00 grep --color=auto 3161

So I think that maybe starting the service as root is the issue, so I shut it down and start it up as the neo4j user.

root@precise64:~# service neo4j-service stop
root@precise64:~# su - neo4j 
neo4j@precise64:~$ service neo4j-service start
Using additional JVM arguments:  -server -XX:+DisableExplicitGC -    Dorg.neo4j.server.properties=conf/neo4j-server.properties -Djava.util.logging.config.file=conf/logging.properties -Dlog4j.configuration=file:conf/log4j.properties -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
Starting Neo4j Server...WARNING: not changing user
process [3435]... waiting for server to be ready....... OK.
http://localhost:7474/ is ready.
cp: cannot create regular file `/var/run/neo4j.pid': Permission denied

And it fails because the pid file cannot be created. Which I assume means this is not the correct approach, but I go ahead and edit /etc/init.d/neo4j-service to change the location of the pid file to the home folder of the neo4j user:

#PIDFILE=/var/run/$NAME.pid
PIDFILE=/var/lib/neo4j/$Name.pid

And then things startup without error when I run the service as the Neo4j user.

So, my question is: What is the intended behavior? Should starting the service as root work as long as the neo4j user has the correct ulimit permissions, or do I need to modify the startup script/permissions to allow the neo4j user to create it's PID file?

Linux Solutions

Solution 1 - Linux

As mentioned in this issue comment and fixed by this commit, since Neo4J 3.1,

> you can add a line in /etc/default/neo4j: > > NEO4J_ULIMIT_NOFILE=60000 > > to set the ulimit setting (60000 open files) for the service.

There is no need anymore to use /etc/security/limits.conf on debian to set the number of open files.

Solution 2 - Linux

You need to add the following entries into the /etc/security/limits.conf file

root   soft    nofile  40000
root   hard    nofile  40000

You need to start the service as root or using sudo

When you do that, the user that ends up starting the service is the root user.

If you dont have an entry in the file for the root user, then it would not work.

So, you just need to add those entries in the file and then reboot your server.

I had the same problem and that is how I was able to resolve it.

Solution 3 - Linux

The solutions posted by @israel and by @Tomasz Swider did not work for me (I am running Ubuntu 14.04). However, I could get rid of this warning by modifying the file /etc/init.d/neo4j-service as suggested by @zwol in the comments below the question:

I added the line ulimit -n 40000 to the do_start() method in this file. In my case this method then looks as follows:

do_start()
{
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        ulimit -n 40000
        start-stop-daemon --chuid ${NEO_USER} --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
                || return 1
        start-stop-daemon --chuid ${NEO_USER} --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
                $DAEMON_ARGS \
                || return 2
}

When I then start the service by using the command

sudo service neo4j-service start

it starts without the warning and returns the following message:

Starting Neo4j Server...WARNING: not changing user
process [9921]... waiting for server to be ready...... OK.
http://localhost:7474/ is ready.

I am not sure whether that is the way to go but as written above, that was the only solution that has worked for me so far to get rid of this warning.

EDIT:

Have not tested it, but seems there is now an easier solution available: see @Schrodinger's'Cat's answer.

Solution 4 - Linux

In the /etc/security/limits.conf file you need to set the limits for the user as whom you will run the neo4j server so if the user you want to use is root, then use root but it is probably bad idea, the settings:

neo4j   soft    nofile  40000 
neo4j   hard    nofile  40000

would work if you where to add neo4j user, I had some problems with that so i just set these to my default user which is 'bob'.

bob   soft    nofile  40000 
bob   hard    nofile  40000

and then I just started the neo4j logged in as bob.

Solution 5 - Linux

I applied the changes as suggested in previous answers

root   soft    nofile  40000
root   hard    nofile  40000

neo4j   soft    nofile  40000 
neo4j   hard    nofile  40000

> /etc/security/limits.conf

Nevertheless I still got the warning about only 1024 open files being allowed. The command

ulimit -a

did show only 1024 open files allowed.

The trick was to log out of the server, then log back in. In the new session the values of the new limits-configuration were applied and I could

service neo4j-server restart

with 40.000 open files allowed.

Solution 6 - Linux

This worked for me on Arch Linux (x86_64), where prior to these modifications ulimit -n was 1024):

Edit /etc/pam.d/su, add:

session required pam_limits.so
Edit both

/etc/systemd/system.conf

/etc/systemd/user.conf

add to each (whatever value you desire: I chose 100,000):

DefaultLimitNOFILE=100000

Reboot.

$ ulimit -n 100000

Based on

Solution 7 - Linux

You can set the hard limits in /etc/security/limits.conf and restart the server to make it into effect.

But with current session, you can run

> ulimit -n 40000

Solution 8 - Linux

Just check once the current value from below command ulimit -n and ulimit -a

And you can edit the value on path below /etc/security/limit.conf

Solution 9 - Linux

The answer in the neo4j community is correct, however what I was missing was that for that to work you should also use neo4j as a service on debian and not just like a simple process. This is also confirmed in the official docs. In my case neo4j start always threw the error.

My current setup that is working:

systemctl edit neo4j.service
insert: LimitNOFILE=40000
start neo4j as a service: systemctl start neo4j.service

I can then check the No. as follows: cat /proc/[processId]/limits

Content Type	Original Author	Original Content on Stackoverflow
Question	Dave Kerber	View Question on Stackoverflow
Solution 1 - Linux	Schrodinger's'Cat	View Answer on Stackoverflow
Solution 2 - Linux	Israel	View Answer on Stackoverflow
Solution 3 - Linux	Cleb	View Answer on Stackoverflow
Solution 4 - Linux	Tomasz Swider	View Answer on Stackoverflow
Solution 5 - Linux	Tobias Gassmann	View Answer on Stackoverflow
Solution 6 - Linux	Victoria Stuart	View Answer on Stackoverflow
Solution 7 - Linux	Vineeth	View Answer on Stackoverflow
Solution 8 - Linux	LetsNoSQL	View Answer on Stackoverflow
Solution 9 - Linux	elod008	View Answer on Stackoverflow