How to install Elasticsearch and Kibana 8.0 on Centos 7

12 min read
How to install Elasticsearch and Kibana 8.0 on Centos 7
Photo by NASA / Unsplash

Intro

Preamble

This blog has been updated for Elastic version 8.0

If you are new to Elasticsearch please read the blog post entitled “What is Elastic and Why Should You Care” it goes into further detail regarding what Elastic is, and how we will use it.

In this tutorial we will bring up a single node cluster for demonstration purposes, the setup we are doing now forms preparatory work to be able to test more advanced features of Elastic in later blog posts.

Don’t forget this is not intended for Production, only as reference material.

This tutorial will focus on Linux as that is where I am most familiar with running Elastic. If there is sufficient demand, I will make a post on how to install and configure it on Windows.

If you don’t understand some of the terms on in this tutorial please see if it’s hyperlinked or in the blog post about Elasticsearch or shoot me an email.

Prerequisites

Required Software·

  • OS Centos 7 (if you plan on running Elasticsearch on a VM be sure to add at least 8GB of RAM and 2 CPU cores See Troubleshooting for advice)
  • VirtualBox
  • Vagrant
  • Elasticsearch 8.0.0
  • Kibana 8.0.0
  • Vi/Vim/Emacs (or your favourite text editing program)
  • Browser (Firefox, Chrome, etc)
  • 8 GB RAM
  • 4 vCPU

Conventions used in this blog post

Code will appear as code
Things to be cautious around are wrapped like this ! Warning ! Be careful and take heed.

Terminals are presented like this:
Windows PowerShell

example

Linux
Root user

example

Unprivileged user

example

There is a small copy button inside the code blocks it looks like this when moused over (It won’t copy hostname information so you can paste with ease)

Lab Diagram

If you open VirtualBox GUI and put it to one side you will get a better feel for how Vagrant is acting as an API to communicate and provision these VMs in VirtualBox. Also very useful for when things don’t go as planned and more advanced troubleshooting is necessary.

Building the Vagrant VM

This section can be skipped if you already have a Centos Linux VM to hand, or are not planning to use a Linux VM. Skip to the configuration section.

If you are new to virtualization please see the blog post “What is Virtualization and Why Should You Care”.

We are going to use VirtualBox and Vagrant to provision a VM for our work. Vagrant provides a platform to provision the same VM again and again, useful for infrastructure as code.

To get Vagrant working you will need a VM provider, we will use VirtualBox but if you already have VMware Workstation then use that instead (you will need to replace the config.vm.provider section in the scripts). Oracle’s VirtualBox-6.1 can be downloaded here. Once it is installed on your host server you need to install Vagrant, download it from here. Vagrant provisions the guest VMs which run on the host server. Once both are installed and configured correctly, we can proceed.

If you are unfamiliar with Vagrant please read the documentation here.

OS

Windows

Provision Centos 7 VM

To provision the VM once VirtualBox and Vagrant have been installed run the following command

The code can be found on my Github here

Replace USERNAME with your username

mkdir -p C:\Users\USERNAME\Boxes\Elastic_Kibana_8.0

Next cd to that DIR

cd C:\Users\USERNAME\Boxes\Elastic_Kibana_8.0

To initialize the box run

vagrant.exe init bento/centos-7

Change the vagrantfile you just created to match this (if you have enough RAM on your host)

notepad.exe .\Vagrantfile

Copy and replace everything in the file with the below

Vagrant.configure("2") do |config|
  config.vm.box = "bento/centos-7"
  config.vm.network "forwarded_port", guest: 5601, host: 5601, host_ip: "127.0.0.1"
  config.vm.provider "virtualbox" do |vb|
    vb.cpus = "4"
    vb.memory = "8192"
  end
end

We are adding in a port_forwarding directive so we can get to the Kibana webpage from the host machine

Bring the Box up 

vagrant.exe up

Log into the box to validate that everything worked thus far

vagrant.exe ssh

The expected result is

vagrant.exe ssh
This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento

[vagrant@localhost ~]$

Linux

Provision Centos 7 VM

To provision the VM once VirtualBox and Vagrant have been installed run the following command

The code can be found on my Github here

Replace USERNAME with your username

mkdir -p /home/USERNAME/Boxes/Elastic_Kibana_8.0

Next cd to that DIR

cd /home/USERNAME/Boxes/Elastic_Kibana_8.0

Pro tip you can use the Linux "bang" command to easily cd into a dir you just made

cd !$

See the links section for more info on Linux bang commands

To initialize the box run

vagrant init bento/centos-7

Change the vagrantfile you just created to match this (if you have enough RAM on your host)

vi Vagrantfile

Copy and replace everything in the file with the below

Vagrant.configure("2") do |config|
  config.vm.box = "bento/centos-7"
  config.vm.network "forwarded_port", guest: 5601, host: 5601, host_ip: "127.0.0.1"
  config.vm.provider "virtualbox" do |vb|
    vb.cpus = "4"
    vb.memory = "8192"
  end
end

We are adding in a port_forwarding directive so we can get to the Kibana webpage from the host machine

Bring the Box up

vagrant up

Log into the box to validate that everything worked thus far

vagrant ssh

The expected result is

vagrant ssh

This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento
[vagrant@localhost ~]$

Install Elasticsearch and Kibana

Once you have logged into your VM we will now add the required repos for Elasticsearch and Kibana (Again this is a Centos Install Guide it won’t work in the current state with Deb based machines)
This method is different from how it is installed if you use the Vagrant bootstrap script

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the Elasticsearch repo to the repo directory (copy until after the last EOF so it can be pasted in one go to the Linux terminal)

sudo bash -c 'cat > /etc/yum.repos.d/elasticsearch.repo' << EOF
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
EOF

Add the Kibana Repo

sudo bash -c 'cat > /etc/yum.repos.d/kibana.repo' << EOF
[kibana-8.x]
name=Kibana repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

Install Elasticsearch, Kibana, and Unzip

sudo yum install --enablerepo=elasticsearch,kibaba elasticsearch kibana unzip

You should see this output from Elasticsearch Note down the elastic superuser password

And this output for Kibana

Configure Elasticsearch and Kibana for Secure Settings

Now that Elastic and Kibana have been installed lets configure them for a secure single node cluster.
Even through Elasticsearch is now "secure by default" we are still going to generate our own certificates this will allow easy setup of HTTPS in Kibana.
Change the below ip:[‘x.x.x.x’] to fit your environment (If you aren’t using the Vagrant VM make sure your IP address isn't a very dynamic DHCP address as you will have problems with SSL Certs later).
The certs will use the IP address as an attribute and if it changes you will need to generate new server certs (Can be done from the CA .crt and .key that will accompany the certs).
Create the cert file. In this instance we are using a single node cluster so we only have two instances with the same IP. We will generate two certificates, one for Elasticsearch and one for Kibana (and a Root CA cert).

Create the DIR

mkdir /tmp/certs

Make the instance.yml file

cat > /tmp/certs/instance.yml << EOF
instances:
  - name: 'elasticsearch'
    dns: [ 'elasticsearch.localdomain']
    ip: [ '10.0.2.15' ]
  - name: 'kibana' 
    dns: [ 'kibana.localdomain' ]
    ip: [ '10.0.2.15' ]
EOF

Generate the certificates

sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --pem --pass secret --out /tmp/certs/elastic-stack-ca.zip

Expected result

Unzip the CA certs

sudo unzip /tmp/certs/elastic-stack-ca.zip -d /tmp/certs/

Generate the server certs with the new CA

sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca-cert /tmp/certs/ca/ca.crt -ca-key /tmp/certs/ca/ca.key --ca-pass secret --pem --in /tmp/certs/instance.yml --out /tmp/certs/certs.zip

Unzip the server certs

sudo unzip /tmp/certs/certs.zip -d /tmp/certs/

Make the dir for Kibana

sudo mkdir /etc/kibana/certs/

Copy the certs into place

! Rename "certs/elasticsearch and certs/kibana" to whatever you named your certs when you made them if you are using a different name !

sudo cp -r /tmp/certs/ca/ca.crt /tmp/certs/elasticsearch/* /etc/elasticsearch/certs
sudo cp -r /tmp/certs/ca/ca.crt /tmp/certs/kibana/* /etc/kibana/certs

Backup the existing certs to root home dir (or wherever else you'd like, you might need the CA files sooner or later)

sudo cp -r /tmp/certs/* /root/

Configure Elasticsearch

Backup existing file

sudo mv /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak

Paste new config into the .yml config file
! Rename "elasticsearch" in the http and ssl section to whatever you named your certs when you made them !
! Change the network.host to whatever you have if it's different !

sudo bash -c 'cat > /etc/elasticsearch/elasticsearch.yml' << EOF
# ======================== Elasticsearch Configuration =========================
#
# ----------------------------------- Paths ------------------------------------
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
# ---------------------------------- Network -----------------------------------
network.host: 10.0.2.15
http.port: 9200
# --------------------------------- Discovery ----------------------------------
discovery.type: single-node
# ---------------------------------- Various -----------------------------------
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
EOF

Add firewall rules (N.B. Not needed if you are using the Vagrant Centos VM)
Add the port and make sure the ssh service is in the list so you don't lose connectivity

sudo firewall-cmd –permanent --add-port=9200/tcp
sudo firewall-cmd –permanent --add-port=5601/tcp

Now we test to see if Elasticsearch will start

sudo systemctl start elasticsearch

If the services fails to start make sure the certificates are ok and in the right place, if you get a service timeout you will need to edit the systemd service file.
Check the certificates

sudo openssl verify -CAfile /etc/elasticsearch/certs/ca.crt /etc/elasticsearch/certs/elasticsearch.crt

Should report OK, if they aren’t found move them from where we made them in /tmp/
Increase the service start timeout

sudo sed -i 's/TimeoutStartSec=75/TimeoutStartSec=500/g' /lib/systemd/system/elasticsearch.service

Reload the service file

sudo systemctl daemon-reload

Validate Elasticsearch

sudo systemctl start elasticsearch

Check it with (Should report back "active (running)")

sudo systemctl status elasticsearch

Now that it’s running we can issue a cURL command to get some info (the password is the elastic users password that was displayed when Elasticsearch was installed)

curl -k -XGET -u elastic https://10.0.2.15:9200/_cat/nodes?pretty

Expected result

Configure Kibana

Set the kibana_system user password
! Do NOT do this in prod !
! Save the password in a secure place !

sudo bash -c '/usr/share/elasticsearch/bin/elasticsearch-reset-password -b -u kibana_system -a > /root/Kibpass.txt'

Add the Kibana password to the keystore

sudo grep "New value:" /root/Kibpass.txt | awk '{print $3}' | sudo /usr/share/kibana/bin/kibana-keystore add --stdin elasticsearch.password

Add a new config file
! Change the “server.ssl.certificate” cert name if you used your hostname !

sudo bash -c 'cat > /etc/kibana/kibana.yml' << EOF
# =========================== Kibana Configuration ============================
# -------------------------------- Network ------------------------------------
server.host: 10.0.2.15
server.port: 5601
# ------------------------------ Elasticsearch --------------------------------
elasticsearch.hosts: ["https://10.0.2.15:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "\${elasticsearch.password}"
# ---------------------------------- Various -----------------------------------
server.ssl.enabled: true
server.ssl.certificate: "/etc/kibana/certs/kibana.crt"
server.ssl.key: "/etc/kibana/certs/kibana.key"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.crt" ]
# ---------------------------------- X-Pack ------------------------------------
xpack.security.encryptionKey: "$(tr -dc A-Za-z0-9 <∕dev∕urandom | head -c 32 ; echo '')"
xpack.encryptedSavedObjects.encryptionKey: "$(tr -dc A-Za-z0-9 <∕dev∕urandom | head -c 32 ; echo '')"
xpack.reporting.encryptionKey: "$(tr -dc A-Za-z0-9 <∕dev∕urandom | head -c 32 ; echo '')"

Don’t worry about the X-Pack settings now they will come into play in a later blog post.
The Vagrant bootstrap.sh script uses the Kibana keystore, this is beyond the scope of this post.
Validate the config

sudo cat /etc/kibana/kibana.yml

Expected result

Start Kibana

sudo systemctl start kibana

Kibana doesn't readily log failures so check it after 30 seconds with

sudo systemctl status kibana

Validate it all comes together by visiting https://127.0.0.1:5601 in your favourite browser on your host machine (Not the headless Vagrant VM, but the machine you ran Vagrant from)
You should see the login page like below (After you added an exception for the self-signed cert)

The elastic users password was the one displayed during the Elasticsearch install
Don't worry if you missed it run this command

sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

If you have made a mistake and do not get the login page at this point you can issue a “vagrant destroy” command and start from the beginning and make sure you copied the Vagrantfile correctly if you followed along with the Vagrant section and have the configs copied correctly.

vagrant.exe destroy
vagrant destroy

Congratulations

Congratulations you now have a demo Elasticsearch + Kibana cluster, in the next blog post we will start to add data. In the mean time have a look around Kibana and it’s features.

You have now built a single node Elasticsearch + Kibana cluster. It won’t do much for you at the moment but watch this space for the next blog posts to see how we can make it work for us.

! Don’t forget this Vagrant setup is not for production !
! Once you have noted all the user/passes down from /root/Kibpass.txt DELETE IT !
! If a nefarious agent gets your Production clusters Elastic password it is GAME OVER !

Automation

If you would like to bring this single node cluster up automatically with all the configs already done please see my GitHub page
“WhY IsN’t It AnSiBlE?!?” The bootstrap.sh provision script is in bash for ease of use and understanding, I would like to make using Elastic as easy as possible. In a later guide we will deploy it with Ansible for “FULL AUTOMATION!”

The readme explains how to bring the VM up with Vagrant

Troubleshooting

Not enough RAM

In the Vagrantfile change this value From vb.memory = "8192" to vb.memory = "6144” or try lower but Elasticsearch doesn't play well with anything less than 8GB

Once the change is made and the file saved reload the VMs with

vagrant.exe reload
vagrant reload

Not enough vCPU cores

You can reduce the CPU count for the Elastic VM in the Vagrantfile to 2 vCPUs (This might slowdown Elasticsearch and Kibana)From vb.cpus = "4" to vb.cpus = "2"

Once the change is made and the file saved reload the VMs with

vagrant.exe reload
vagrant reload

Linux bang commands https://www.redhat.com/sysadmin/bash-bang-commands(protip)

Elasticsearch 8.0 install guide https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html (very useful)

Kibana 8.0 install guide https://www.elastic.co/guide/en/kibana/current/rpm.html (very useful)

SSL creation https://www.golinuxcloud.com/enable-https-ssl-secure-elasticsearch-linux/(very useful)

SSL creation https://www.elastic.co/guide/en/elasticsearch/reference/7.14/security-basic-setup-https.html#encrypt-kibana-elasticsearch(Kind of useful)

Nodes https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html(Useful)

Why we add the X-Pack settings in the Kibana Config https://www.elastic.co/guide/en/kibana/8.0/xpack-security-secure-saved-objects.html(Useful)

Dylan Shield

Dylan Shield

More posts
Previous Post
Next Post