Elasticsearch: Max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
ElasticsearchLimitSystemdCgroupsElasticsearch Problem Overview
I have an issue with a systemd config for ElasticSearch.
[Unit]
Description=platform-elasticsearch
After=syslog.target network.target remote-fs.target nss-lookup.target
[Service]
User={{ app_user }}
Group={{ app_group }}
Environment=ES_PATH_CONF=/platform/opt/elasticsearch-{{ elasticsearch.version }}/config
Environment=JAVA_HOME=/platform/opt/jdk{{ jdk.major_version }}_{{ jdk.minor_version }}
LimitAS=infinity
LimitRSS=infinity
LimitCORE=infinity
LimitNOFILE=100000
LimitMEMLOCK=100000
StandardOutput=syslog
StandardError=syslog
WorkingDirectory=/platform/var/app/elasticsearch
ExecStart=/platform/opt/elasticsearch-{{ elasticsearch.version }}/bin/elasticsearch
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s -TERM $MAINPID
TimeoutStopSec=60
# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143 0
Type=simple
Restart=on-failure
RestartSec=10
PIDFile=/platform/var/run/elasticsearch.pid
[Install]
WantedBy=multi-user.target
This does not seem to let me configure the vm.max_map_count setting.
Jul 20 14:53:46 scratchpad elasticsearch: [2018-07-20T14:53:46,359][INFO ][o.e.b.BootstrapChecks ] [1oQJNUK] bound or publishing to a non-loopback address, enforcing bootstrap checks
Jul 20 14:53:46 scratchpad elasticsearch: ERROR: [1] bootstrap checks failed
Jul 20 14:53:46 scratchpad elasticsearch: [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Jul 20 14:53:46 scratchpad elasticsearch: [2018-07-20T14:53:46,376][INFO ][o.e.n.Node ] [1oQJNUK] stopping ...
Jul 20 14:53:46 scratchpad elasticsearch: [2018-07-20T14:53:46,414][INFO ][o.e.n.Node ] [1oQJNUK] stopped
Jul 20 14:53:46 scratchpad elasticsearch: [2018-07-20T14:53:46,414][INFO ][o.e.n.Node ] [1oQJNUK] closing ...
Jul 20 14:53:46 scratchpad elasticsearch: [2018-07-20T14:53:46,445][INFO ][o.e.n.Node ] [1oQJNUK] closed
Jul 20 14:53:46 scratchpad systemd: platform-elasticsearch.service: main process exited, code=exited, status=78/n/a
The specific issue is the following:
Jul 20 14:53:46 scratchpad elasticsearch: [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
I have been able to start elastic search on the commandline with the following:
sudo su -c 'echo 262144 > "/proc/sys/vm/max_map_count"' && \
export JAVA_HOME=/platform/opt/jdk1.8.0_181 && \
export ES_PATH_CONF=/platform/opt/elasticsearch-6.3.1/config && \
/platform/opt/elasticsearch-6.3.1/bin/elasticsearch
can anyone tell me why LimitMEMLOCK=100000 does not work, and how I can effectively set max_map_count from within systemd.
I have also tried to set the following:
cat /etc/security/limits.d/30_elastic_limits.conf
vagrant hard nofile 500000
vagrant hard memlock 262144
but this seems to be totally ignored by systemd.
Elasticsearch Solutions
Solution 1 - Elasticsearch
sysctl -w vm.max_map_count=262144
is correct, however, the setting will only last for the duration of the session. If the host reboots, the setting will be reset to the original value.
If you want to set this permanently, you need to edit /etc/sysctl.conf
and set vm.max_map_count
to 262144.
When the host reboots, you can verify that the setting is still correct by running sysctl vm.max_map_count
Solution 2 - Elasticsearch
for windows users, using wsl subsystem
open powershell run
wsl -d docker-desktop
then
sysctl -w vm.max_map_count=262144
Solution 3 - Elasticsearch
Insert the new entry into the /etc/sysctl.conf file with the required parameter:
vm.max_map_count = 262144
it make changes permanent.
Also run:
sysctl -w vm.max_map_count=262144
change current state of kernel.
If you use docker to take effect you should restart it:
systemctl restart docker
Solution 4 - Elasticsearch
See the Elasticsearch documentation about virtual memory. On Centos you can do with following command:
sysctl -w vm.max_map_count=262144
Solution 5 - Elasticsearch
This is not an answer per se but a clarification/shortcut for anyone having the op's problem from a docker container perspective. I had this problem from an application running in a docker container. And as explained here by nishant
You don’t need to increase the virtual memory for Elasticsearch at the container level, you can do it for the host machine by running this command:
sudo sysctl -w vm.max_map_count=262144
and then restart your docker-containers.
As explained by val above setting this max_map_count
this way won't persist upon the restart of the machine on which is running the docker container. and so you will need to save it in a more persistent manner as explained by him above.
Solution 6 - Elasticsearch
You can also resolve the memory constraint issue by using single-node
discovery type. Set this in the environment: discovery.type=single-node
docker-compose.yml
services:
es:
image: elasticsearch
environment:
- discovery.type=single-node
See also:
Solution 7 - Elasticsearch
I ran into the same issue for the single node elastic search cluster. As perelastic-search documentation, for running single node, you have use "discovery.type=single-node" in the docker run command.
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.13.3
In docker-compose.yml, you can specif below:
version: '3.1'
services:
elastic_search:
image: docker.elastic.co/elasticsearch/elasticsearch:7.13.3
container_name: es01
environment:
- discovery.type=single-node
- node.name=es01
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ports:
- 9200:9200
networks:
- elastic
networks:
elastic:
Solution 8 - Elasticsearch
Please run the following command: sysctl -w vm.max_map_count=262144
to increase the default virtual memory used by the Elasticsearch.
Note: When you run the above-mentioned command your problem will get resolved but, this will be a temporary solution as node/system/container gets restart your changes will go. So if you want to set this permanently, you need to edit /etc/sysctl.conf
and set vm.max_map_count
to 262144
.
For more detail click here.
Solution 9 - Elasticsearch
Max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144].
Please run the command below to fix this issue:
sysctl -w vm.max_map_count=262144
Solution 10 - Elasticsearch
vm.max_map_count=262144
Add this line in to below path
vim /etc/sysctl.conf
or go to this link https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
Solution 11 - Elasticsearch
For WSL, you can create config file (.wslconfig) at:
C:\Users\[your user]\.wslconfig
and put in it:
[wsl2]
kernelCommandLine = "sysctl.vm.max_map_count=262144"
So that you do not have to set max_map_count
every time you restart PC
Solution 12 - Elasticsearch
In case anyone is on elasticsearch 8.0 with kubernetes ECK operator.
Set runAsUser: 0
elastic/cloud-on-k8s#5410 (comment)
>Starting with 8.0 Elasticsearch container is not running as root anymore, which is required to run sysctl. You can add runAsUser: 0 to force the init container to run as root