How to log all executed elasticsearch queries

I want to see all queries executed against an elasticsearch instance. Is it possible to run elasticsearch in a debug mode, or to tell it to store all queries executed against it?

The purpose is to see which queries are launched from a software using elasticsearch for analysis.

In versions of ElasticSearch prior to 5, you can accomplish this by changing the ElasticSearch.yml configuration file. At the very bottom of this file, you can adjust the logging time to record all:

index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms

index.search.slowlog.threshold.fetch.warn: 1s  
index.search.slowlog.threshold.fetch.info: 800ms
index.search.slowlog.threshold.fetch.debug: 500ms
index.search.slowlog.threshold.fetch.trace: 200ms

index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
index.indexing.slowlog.threshold.index.trace: 500ms

Adjust the settings and restart your node, then consulting the logs to view the queries executed against your node. Note if in production log files will rapidly increase in size.

In version 5.x, you have to set slow log logging per index.

Command line:

curl -XPUT 'http://localhost:9200/myindexname/_settings' -d '{
"index.indexing.slowlog.threshold.index.debug" : "0s",
"index.search.slowlog.threshold.fetch.debug" : "0s",
"index.search.slowlog.threshold.query.debug" : "0s"
}'

Or, if you are using Kibana, go to the Dev Tools bar and enter:

PUT /myindexname/_settings 
{"index.indexing.slowlog.threshold.index.debug": "0s", 
"index.search.slowlog.threshold.fetch.debug" : "0s", 
"index.search.slowlog.threshold.query.debug": "0s"}

#1: Apply to ALL indices

You can apply the setting to ALL indices with the following command:

PUT /_all/_settings 
{"index.indexing.slowlog.threshold.index.debug": "0s", 
"index.search.slowlog.threshold.fetch.debug" : "0s", 
"index.search.slowlog.threshold.query.debug": "0s"}

#2: Preserve existing settings

If you don't want to overwrite existing settings, but just add new ones, add '''preserve_existing=true''' after _settings, like this:

PUT /_all/_settings?preserve_existing=true 
{"index.indexing.slowlog.threshold.index.debug": "0s", 
"index.search.slowlog.threshold.fetch.debug" : "0s", 
"index.search.slowlog.threshold.query.debug": "0s"}

The above request will ONLY add the settings if they don't exist. It will not change them if they are already there.

#3: All available log settings

All available slow log settings are here and below for your reference:

PUT /test_index/_settings
{
"index.search.slowlog.threshold.query.warn": "60s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "1s",
"index.search.slowlog.threshold.query.trace": "0.1s",
"index.search.slowlog.threshold.fetch.warn": "30s",
"index.search.slowlog.threshold.fetch.info": "5s",
"index.search.slowlog.threshold.fetch.debug": "1s",
"index.search.slowlog.threshold.fetch.trace": "0.1s",
"index.indexing.slowlog.threshold.index.warn": "6s",
"index.indexing.slowlog.threshold.index.info": "5s",
"index.indexing.slowlog.threshold.index.debug": "1s",
"index.indexing.slowlog.threshold.index.trace": "0.1s",
"index.indexing.slowlog.level": "info",
"index.indexing.slowlog.source": "1000"
}

Starting with Version 5 ElasticSearch charges money for this functionality. It's called "Audit log" and is now part of X-Pack. There is a basic license available that is free, but this license only gives you a simplistic monitoring functionality. Authentication, query logging and all these rather basic things cost money now.

Yes, it's possible to tell Elasticsearch to log all queries executed against it and you can configure logging levels, such as DEBUG. You can change it in ES 7.13.x using curl:

curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "transient": {
    "logger.org.elasticsearch.discovery": "DEBUG"
  }
}
'

On macOS log files are stored on $ES_HOME by default. Please check the docs about Elasticsearch Logging