How to access Kibana from Amazon elasticsearch service?

I created Amazon elasticsearch service and populated data into it using logstash, which has been installed on an EC2 instance. On the Amazon elasticservice console page, there will be a link to access Kibana.

search-cluster_name-XXXXXXXXXXXXXXXXXXX.region_name.es.amazonaws.com/_plugin/kibana/

when I click the link, browser is throwing the following error.

{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: arn:aws:es:region_name:account_id:domain/cluster_name/_plugin/kibana/"}

I'm sure that this has something related with access policy of ES domain.How should I modify my access policy so that I can access Kibana from a click on the link specified ?

You can setup an Access Policy with both IAM and IP-address based access. See my answer here. In short:

• EC2 instance needs a profile with the arn:aws:iam::aws:policy/AmazonESFullAccess policy
• Policy should include two statements: first list IAM access, second list IP access.

Here's an example policy (statement order is important!)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "192.168.1.0",
            "192.168.1.1"
          ]
        }
      }
    }
  ]
}

I used for that purpose proxy tool called aws-es-kibana. It signs all your requests sent to aws kibana.

IAM configuration:

I created new IAM user "elasticsearch_user" with programmatic access (and I got accessKeyId and secretAccessKey associated with that account).

Elasticsearch configuration:

I created elasticsearch policy that enables access for the new created IAM user:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::{YOUR_AWS_ACCOUNT_ID}:user/elasticsearch_user"
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-central-1:{YOUR_AWS_ACCOUNT_ID}:domain/{YOUR_ELASTICSEARCH_DOMAIN}/*"
    }
  ]
}

Connect to kibana from your local station:

To connect from my local station (windows) to kibana I just need to type in console:

SET AWS_ACCESS_KEY_ID=myAccessKeyId
SET AWS_SECRET_ACCESS_KEY=mySecretAccessKey

aws-es-kibana search-{PROTECTED_PART_OF_YOUR_ELASTICSEARCH_ENDPOINT}.eu-central-1.es.amazonaws.com

After that you should have proxied access to your kibana under: http://127.0.0.1:9200/_plugin/kibana

You have to configure an access policy for your elasticsearch cluster. there are two options:

1. Set up an IAM-based access policy
2. White-list certain IPs from which people can access your Kibana instance.

Option 1, using IAM based access is the better option:

• Create an IAM user, called kibana_user with programmatic access. Save the accessKeyId and the secretAccessKey. Also copy the user's ARN.
• Configure your access policy to give access to kibana_user.
  • Go to https://eu-central-1.console.aws.amazon.com/es/
  • Select your elasticsearch domain
  • Click on "Modify access policty"
  • Click on "Select a template" and use the one that's called "Allow access to one or more AWS accounts or IAM users". Enter the ARN of the kibana_user
• Unfortunately, AWS does not provide with a way to log in as that user and then connect to Kiabana. Instead, if wants you to sign the HTTP requests that you make to Kibana with that user's key. There are tools that do this for you, for example aws-es-proxy

I seriously recommend against the second option with IP-based access. Even if you have a static IP,

• everybody on that IP will have access to your data on elasticesarch
• you only have access if you are connected via that API. Not from your phone, not from home.

The only case where this makes sense is if you are running your own proxy server with its own authentication method and a static IP.

In my case, I had an nginx server running which already had access to the elasticsearch service. So all I had to do was to add a proxy on this nginx. No changes in AWS IAM required.

Add this to /etc/nginx/sites-enabled/elasticsearch

server {
  listen   7777;
  server_name  127.0.0.1 default_server;
  access_log  /var/log/nginx/elasticsearch.access.log;
  location / {
    auth_basic "My Super Secret Server";
    auth_basic_user_file /etc/nginx/.elasticsearch_htpasswd;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   Host      $http_host;
        proxy_pass         https://<your_server_here>.es.amazonaws.com/;
        proxy_set_header Authorization "";
        proxy_hide_header Authorization;
  }
}

and restart nginx. Then you can access kibana at:

http://your_nginx_server_name.com:7777/_plugin/kibana/app/kibana#/dev_tools/console?_g=()

The file /etc/nginx/.elasticsearch_htpasswd is a standard apache2 htaccess file. You can find more about basic auth for nginx here.

NOTE: Basic auth is NOT a recommended way to secure anything. Definitely don't use this in production.

You may need to have IP-based policy and allow access to your domain from specific IP (Kibana's).

Other option (aside from changing access policy to be completely open) would be signing requests - IIRC this helped a friend of mine with similar message.

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-signing-service-requests

See also "Connecting a Local Kibana Server to Amazon Elasticsearch Service" on the same page.