Start. Proposed – ELK

powerful, flexible, fairly easy to handle tool for collecting, storing and visual data analysis

The starting point for the project was the treatment of the company address from the client, which is a major infrastructure provider. The client on the basis of the analysis of the data, representing logs, optimizes the capacity of hardware and software. Naturally I had to transfer this task to a higher level with increasing volume of information. In other words it took a powerful, flexible, fairly easy to handle tool for collecting, storing and visual data analysis. As such a tool we have developed a stack of software products under the name ELK.

ELK stands for Elasticsearch, Logstash and Kibana. It’s essentially three full-fledged independent software open source product, combined into one powerful solution for a wide range of tasks for processing data.

The objectives and performance requirements

To deploy a monitoring system for:

  1. capacity planning (finding bottlenecks hardware and software)
  2. retrospective monitoring of events from end user equipment

The system shall meet the following performance requirements:

  • Up to 1000 events/sec
  • The growth data up to 3GB/hour
  • Storage up to 90 days
  • Collecting 95% of events over a period of time up to 24 hours (displayed by default)
  • Collecting 95% of events with an interval of 5 min (display default)
  • Connection up to 10 clients HTTP
  • Fault tolerance 1 server in the cluster (replication 2)
  • On servers running ELK only (no third party apps)

Requirements for hardware and software systems:

  • 3 servers
  • OS – 7.2 CentOS / RHEL 7.2
  • 8 physical CPU cores on the server (or more according to physical cores without redistribution between the VM)
  • 48GB RAM on the server (without redistribution between RAM VM)
  • 8TB HDD to the server, providing, at least, the speed of random I / o-output 500 IOPS (70% read / 30% write)
  • At least 1Gb network connection between servers

Specification physical servers for deploying the system:

  • 2xE5-2630v4 10c (or E5-2660v4 14c for higher productivity)
  • 4x32GB RAM DDR4
  • 12G 2x300GB SAS 10K (RAID1)
  • 2xPSU

System deployment

On each server node takes place individual instances of Elasticsearch, Logstash and Kibana.

Further, the order of deployment will be considered on the same node.

All the details are applicable to other machines in the cluster.

PE

1. Stack of Elasticsearch, Logstash and Kibana (ELK) is set from one repository. Create a system description file of the repository:

2. Import Elasticsearch PGP Key:

3. Work for Elasticsearch and Logstash you need to download and install JDK 8.

Install Elasticsearch

1. Install Elasticsearch

2. Edit the Elasticsearch configuration:

In the section “Paths” define the path to the data directory and the logs to Elasticsearch:

To improve the stability of the site, excluding unloading of the JVM on a swap partition, raskomentiruyte line bootstrap.memory_lock true section of Memory.
In the section “Network” inserted row for CORS requests:

In the section “Various” optimize resource-intensive operations the following parameters (8 cores CPU):

3. Editable parameters JVM:

Set minimum and maximum JVM heap size parameters Xms and Xmx, respectively. In our case, these values are the same and equal to 32g:

To free unused memory selectable Garbage Collection. In our case we use G1GC. With this purpose in corresponding section changes:

4. Run and install the Elasticsearch startup at boot system:

5. Check the performance of Elasticsearch node using a simple request HTTP

Install Kibana

1. Install Kibana:

2. Edit the Kibana configuration:

Make possible the remote connection to the server Kibana:

URL for querying an instance of Elasticsearch:

Define file embed Kibana logs:

3. Run and install Kibana startup at boot system:

Installing Logstash

1. Install Logstash:

2. Edit configuration Logstash:

In the section “Pipeline Settings” define parameters:

based on the number of CPU cores on a node (8)

based on the number of Elasticsearch instances (3)

3. Edit configuration JVM:

Set minimum and maximum JVM heap size parameters Xms and Xmx, respectively. In our case, these values are the same and equal to 4g:

As for Elasticsearch from select G1GC Garbage Collection. With this purpose in corresponding section changes:

4. Create a config file:

In the input section, for our case, we prescribe the path to the directory containing the log files. In the output pane, specify the Elasticsearch cluster.

5. Run and install the Logstash startup at boot system:

Installed X-Pack as an extension to ELK

X-Pack is a PlugIns to Elasticsearch and Kibana. Includes modules, flexible distribution of access to resources (X-Pack Security), additional graphic information, notification, monitoring, reporting at the server level EK, gcd and indexes in Kibana. Setting up authentication as a native level and using LDAP and AD.

1. Installed X-Pack for Elasticsearch on each node:

2. To automatically generate indices of the X-Pack added the following line to elasticsearch.yml:

3. Restart Elasticsearch:

4. Install X-Pack for Kibana on each node:

5. Restart Kibana:

To disable modules, X-Pack, put the following lines in elasticsearch.yml:

in kibana.yml:

Restart Elasticsearch and Kibana.
To re-enable the commented line and restart the services.

Configure system for use in the cluster

Editable on each node of the configuration file Elasticsearch:

1. In the section “Network” uncomment the appropriate line and set the node address assigned to it in the cluster. For each node its value. For example:

2. In the section “ Cluster” raskomentiruyte the appropriate line and set the unique network name of the cluster. For example:

3. In the section “ Node” uncomment the relevant line prescribe and unique for our cluster the name node. For each node its value. For example:

4. In the section “ Discovery” raskomentiruyte the appropriate line and set ‘ the node, which should be detected in the cluster. For example:

In addition, to avoid splitting our cluster into two independent failure of the system, you must set the following parameters (in our case, 3 nodes):

5. Save changes of the configuration file. Restart Elasticsearch:

Check the cluster status:

Interacting with an Elasticsearch cluster

To interact with the Elasticsearch cluster using the app “elasticsearch-head”. In previous versions of Elasticsearch this application is installed as a plugin.
Starting with version 5.X developers recommend to use the X-Pack to replace the “es-head”. So here had to use “es-head” in the form of a separate web application.
To ensure the primitive web server written in golang.

Create a Dashboard in Kibana

Open the browser to the web interface Kibana (http://ip_addr_kibana:5601). Define the index name or pattern of names of several indexes (index-pattern*). Select the index field with the time stamp, which we will use to compare the time parameters. Clicking “Create”, add the indices. If the index is greater than one, you need to choose one by default. On the Discover page, we get interactive access to the documents of the selected indexes. If the selected index field with the time stamp, the document rendering is complemented by the histogram.
Next, create visualizations and include them in dashboards.
One index created dashboards, including 13 visualizations of various types, in accordance with the specification of the customer.
By means of the module “X-Pack Security” additionally created a role for a user with rights only for viewing (excluding viewing role and user).

In /etc/kibana/kibana.yml defined by the application running for all users after login to server Kibana, the relevant string:

kibana.defaultAppId: “dashboard/Dashboard_Name_Default”

Thank you!

Thank you for viewing this article.

If You have any questions-please email – will be happy to answer!