Cleaning elasticsearch with curator

My article on installing and configuring the log collection system based on the ELK Stack was not complete without one important section. Today I will tell you how to configure automatic cleaning of elasticsearch indexes using curator. Everyone who will operate the system will face this, as it is very demanding of resources. Clearing old indexes increases speed.

[toc]

The article will be short, as the process of cleaning indexes in elasticsearch using curator in the basic version is very simple.

Install curator

First, install the curator. This can be done in different ways. The easiest one is from the packages.elastic.co repository from the product authors. Connect it to CentOS 7.

# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
# mcedit /etc/yum.repos.d/curator.repo

[curator-5]
name=CentOS/RHEL 7 repository for Elasticsearch Curator 5.x packages
baseurl=https://packages.elastic.co/curator/5/centos/7
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Connect repository in Debian 8 / Ubuntu

# wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# mcedit /etc/apt/sources.list.d/curator.list

deb [arch=amd64] https://packages.elastic.co/curator/5/debian stable main

Connect repository in Debian 9

# deb [arch=amd64] https://packages.elastic.co/curator/5/debian9 stable main

Install curtator:

# yum install elasticsearch-curator
# apt update && apt install elasticsearch-curator

Also curator can be installed via pip. For Debian / Ubuntu, just run:

# apt install python-pip

Install curator via pip:

# pip install elasticsearch-curator

Configuring curator for cleaning elasticsearch

For example, let’s make a simple task to close and delete indexes with the nginx- * template older than 14 days. To do this, create a directory for the curator configs and the configs themselves.

# mkdir /etc/curator
# touch /etc/curator/action.yml
# touch /etc/curator/config.yml

Fill the files with the following content. First, a common config.

# mcedit /etc/curator/config.yml

client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Next file with the necessary actions:

# mcedit /etc/curator/action.yml

actions:
  1:
    action: close
    description: >-
      Close indices older than 14 days (based on index name).
    options:
      ignore_empty_list: True
      delete_aliases: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: nginx-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 14

  2:
    action: delete_indices
    description: >-
      Delete indices older than 14 days (based on index name).
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: nginx-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 14

Pay attention to the formatting of the file. The indents at the beginning of the line are important. They should be exactly like my example.

The config is made on the basis of examples from official documentation. I recommend to look for all the details there. We start cleaning:

# /usr/local/bin/curator --config /etc/curator/config.yml /etc/curator/action.yml

In the console you will see an informative output of executable commands for cleaning indexes. After completing the cleanup, do not forget to add the job to cron.

# crontab -e

4 4 * * * /usr/local/bin/curator --config /etc/curator/config.yml /etc/curator/action.yml

Index cleaning will be done every day at 4am.