How to generate a JSON log from nginx?

JsonLoggingNginx

Json Problem Overview


I'm trying to generate a JSON log from nginx.

I'm aware of solutions like this one but some of the fields I want to log include user generated input (like HTTP headers) which need to be escaped properly.

I'm aware of the nginx changelog entries from Oct 2011 and May 2008 that say:

*) Change: now the 0x7F-0x1F characters are escaped as \xXX in an
   access_log.
*) Change: now the 0x00-0x1F, '"' and '\' characters are escaped as \xXX
   in an access_log.

but this still doesn't help since \xXX is invalid in a JSON string.

I've also looked at the HttpSetMiscModule module which has a set_quote_json_str directive, but this just seems to add \x22 around the strings which doesn't help.

Any idea for other solutions to log in JSON format from nginx?

Json Solutions


Solution 1 - Json

Finally it looks like we have good way to do this with vanilla nginx without any modules. Just define:

log_format json_combined escape=json
  '{'
    '"time_local":"$time_local",'
    '"remote_addr":"$remote_addr",'
    '"remote_user":"$remote_user",'
    '"request":"$request",'
    '"status": "$status",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"request_time":"$request_time",'
    '"http_referrer":"$http_referer",'
    '"http_user_agent":"$http_user_agent"'
  '}';

Note that escape=json was added in nginx 1.11.8. http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format

Solution 2 - Json

You can try to use that one https://github.com/jiaz/nginx-http-json-log - addition module for Nginx.

Solution 3 - Json

You can try to use:

PS: The if parameter (1.7.0) enables conditional logging. A request will not be logged if the condition evaluates to “0” or an empty string:

map $status $http_referer{
    ~\xXX  0;
    default 1;
}

access_log /path/to/access.log combined if=$http_referer;

It’s a good idea to use a tool such as https://github.com/zaach/jsonlint to check your JSON data. You can test the output of your new logging format and make sure it’s real-and-proper JSON.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionJules OlléonView Question on Stackoverflow
Solution 1 - JsonpvaView Answer on Stackoverflow
Solution 2 - JsonSeva PoliakovView Answer on Stackoverflow
Solution 3 - JsonValeriy SolovyovView Answer on Stackoverflow