My article on installing and configuring the log collection system based on the ELK Stack was not complete without one important section. Today I will tell you how to configure automatic cleaning of elasticsearch indexes using curator. Everyone who will operate the system will face this, as it is very demanding of resources. Clearing old indexes increases speed.
The article will be short, as the process of cleaning indexes in elasticsearch using curator in the basic version is very simple.
First, install the curator. This can be done in different ways. The easiest one is from the packages.elastic.co repository from the product authors. Connect it to CentOS 7.
# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch # mcedit /etc/yum.repos.d/curator.repo
[curator-5] name=CentOS/RHEL 7 repository for Elasticsearch Curator 5.x packages baseurl=https://packages.elastic.co/curator/5/centos/7 gpgcheck=1 gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch enabled=1
Connect repository in Debian 8 / Ubuntu
# wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - # mcedit /etc/apt/sources.list.d/curator.list
deb [arch=amd64] https://packages.elastic.co/curator/5/debian stable main
Connect repository in Debian 9
# deb [arch=amd64] https://packages.elastic.co/curator/5/debian9 stable main
# yum install elasticsearch-curator # apt update && apt install elasticsearch-curator
Also curator can be installed via pip. For Debian / Ubuntu, just run:
# apt install python-pip
Install curator via pip:
# pip install elasticsearch-curator
Configuring curator for cleaning elasticsearch
For example, let’s make a simple task to close and delete indexes with the nginx- * template older than 14 days. To do this, create a directory for the curator configs and the configs themselves.
# mkdir /etc/curator # touch /etc/curator/action.yml # touch /etc/curator/config.yml
Fill the files with the following content. First, a common config.
# mcedit /etc/curator/config.yml
client: hosts: - 127.0.0.1 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: ['elasticsearch', 'urllib3']
Next file with the necessary actions:
# mcedit /etc/curator/action.yml
actions: 1: action: close description: >- Close indices older than 14 days (based on index name). options: ignore_empty_list: True delete_aliases: False disable_action: False filters: - filtertype: pattern kind: prefix value: nginx- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 14 2: action: delete_indices description: >- Delete indices older than 14 days (based on index name). options: ignore_empty_list: True disable_action: False filters: - filtertype: pattern kind: prefix value: nginx- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 14
Pay attention to the formatting of the file. The indents at the beginning of the line are important. They should be exactly like my example.
The config is made on the basis of examples from official documentation. I recommend to look for all the details there. We start cleaning:
# /usr/local/bin/curator --config /etc/curator/config.yml /etc/curator/action.yml
In the console you will see an informative output of executable commands for cleaning indexes. After completing the cleanup, do not forget to add the job to cron.
# crontab -e
4 4 * * * /usr/local/bin/curator --config /etc/curator/config.yml /etc/curator/action.yml
Index cleaning will be done every day at 4am.