Intro
Preamble
If you are new to Elasticsearch please read the blog post entitled “What is Elastic and Why Should You Care” it goes into further detail regarding what Elastic is, and how we will use it.
In this tutorial we will bring up a single node cluster for demonstration purposes, the setup we are doing now forms preparatory work to be able to test more advanced features of Elastic in later blog posts.
Don’t forget this is not intended for Production. Only as reference material.
This tutorial will focus on Linux as that is where I am most familiar with running Elastic. If there is sufficient demand, I will make a post on how to install and configure it on Windows.
If you don’t understand some of the terms on in this tutorial please see if it’s hyperlinked or in the blog post about Elasticsearch.
Prereqisites
Required Software
· OS (Centos 7 if you plan on running Elasticsearch on a VM be sure to add at least 8GB of RAM and 2 CPU cores)
· VirtualBox
· Vagrant
· Elasticsearch 7.15.0
· Kibana 7.15.0
· Vi/Vim/Emacs (or your favourite text editing program)
· Browser (Firefox, Chrome, etc)
· At least 8GB of RAM See Troubleshooting for advice
Conventions used in this blog post
Code will appear as code
Things to be cautious around are wrapped like this ! Warning ! Be careful and take heed.
Terminals are presented like this;
Windows PowerShell
example
Linux
Root user
example
Unprivileged user
example
There is a small copy button inside the code blocks it looks like this when moused over (It won’t copy hostname information so you can paste with ease)

Lab Diagram

If you open VirtualBox GUI and put it to one side you will get a better feel for how Vagrant is acting as an API to communicate and provision these VMs in VirtualBox. Also very useful for when things don’t go as planned and more advanced troubleshooting is necessary.
Building the Vagrant VM
This section can be skipped if you already have a Centos Linux VM to hand, or are not planning to use a VM. Skip to the configuration section.
If you are new to virtualization please see the blog post “What is Virtualization and Why Should You Care”.
We are going to use VirtualBox and Vagrant to provision a VM for our work. Vagrant provides a platform to provision the same VM again and again, useful for infrastructure as code.
To get Vagrant working you will need a VM provider, we will use VirtualBox but if you already have VMware Workstation then use that instead (you will need to replace the config.vm.provider section in the scripts). Oracle’s VirtualBox-6.1 can be downloaded here. Once it is installed on your host server you need to install Vagrant, download it from here. Vagrant provisions the guest VMs which run on the host server. Once both are installed and configured correctly, we can proceed.
If you are unfamiliar with Vagrant please read the documentation here.
OS
Windows
Provision Centos 7 VM
To provision the VM once VirtualBox and Vagrant have been installed run the following command
The code can be found on my Github here
Replace USERNAME with your username
mkdir -p C:\Users\USERNAME\Boxes\Elastic_Kibana
Next cd to that DIR
cd C:\Users\USERNAME\Boxes\Elastic_Kibana
To initialize the box run
vagrant.exe init bento/centos-7
Change the vagrantfile you just created to match this (if you have enough RAM on your host)
notepad.exe .\Vagrantfile
Copy and replace everything in the file with the below
Vagrant.configure("2") do |config|
config.vm.box = "bento/centos-7"
config.vm.network "forwarded_port", guest: 5601, host: 5601, host_ip: "127.0.0.1"
config.vm.provider "virtualbox" do |vb|
vb.cpus = "4"
vb.memory = "8192"
end
end
We are adding in a port_forwarding directive so we can get to the Kibana webpage from the host machine
Bring the Box up
vagrant.exe up
Log into the box to validate that everything worked thus far
vagrant.exe ssh
The expected result is
vagrant.exe ssh
This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento
[vagrant@localhost ~]$
Linux
Provision Centos 7 VM
To provision the VM once VirtualBox and Vagrant have been installed run the following command
The code can be found on my Github here
Replace USERNAME with your username
mkdir -p /home/USERNAME/Boxes/Elastic_Kibana
Next cd to that DIR
cd /home/USERNAME/Boxes/Elastic_Kibana
To initialize the box run
vagrant init bento/centos-7
Change the vagrantfile you just created to match this (if you have enough RAM on your host)
vi Vagrantfile
Copy and replace everything in the file with the below
Vagrant.configure("2") do |config|
config.vm.box = "bento/centos-7"
config.vm.network "forwarded_port", guest: 5601, host: 5601, host_ip: "127.0.0.1"
config.vm.provider "virtualbox" do |vb|
vb.cpus = "4"
vb.memory = "8192"
end
end
We are adding in a port_forwarding directive so we can get to the Kibana webpage from the host machine
Bring the Box up
vagrant up
Log into the box to validate that everything worked thus far
vagrant ssh
The expected result is
vagrant ssh
This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento
[vagrant@localhost ~]$
Install Elasticsearch and Kibana
Once you have logged into your VM we will now add the required repos for Elasticsearch and Kibana (Again this is a Centos Install Guide it won’t work in the current state with Deb based machines)
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Add the Elasticsearch repo to the repo directory (copy until after the last EOF and it can be pasted in one to the Linux terminal)
sudo bash -c 'cat > /etc/yum.repos.d/elasticsearch.repo' << EOF
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
EOF
Add the Kibana Repo
sudo bash -c 'cat > /etc/yum.repos.d/kibana.repo' << EOF
[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
The way we get packages differs from the Github code.
Install Elasticsearch, Kibana, and Unzip
sudo yum install elasticsearch kibana unzip
Validate that the installs have worked
sudo head -n 3 /etc/elasticsearch/elasticsearch.yml
sudo head -n 3 /etc/kibana/kibana.yml
Expected result

If there is no text the installation failed please start from the beginning and/or check your distro (It is different for RedHat/Debian based Linux distributions, this install guide is only for Centos 7 which is RPM based)
Configure Elasticsearch and Kibana for Secure Settings
Now that Elastic and Kibana have been installed lets configure them for a secure single node cluster.
Change the below ip:[‘x.x.x.x’] to fit your environment (If you aren’t using the Vagrant VM make sure your IP address isn't a very dynamic DHCP address as you will have problems with SSL Certs later)
The certs will use the IP address as an attribute and if it changes you will need to generate new server certs (Can be done from the CA .crt and .key that will accompany the certs)
You have been warned.
Create the cert file. In this instance we are using a single node cluster so we only have two instances with the same IP.
We will generate two certificates, one for Elasticsearch and one for Kibana (and a Root CA cert)
Create the DIR
mkdir /tmp/certs
Make the instance.yml file
cat > /tmp/certs/instance.yml << EOF
instances:
- name: 'elasticsearch'
dns: [ 'elasticsearch.localdomain']
ip: [ '10.0.2.15' ]
- name: 'kibana'
dns: [ 'kibana.localdomain' ]
ip: [ '10.0.2.15' ]
EOF
Generate the certificates
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --keep-ca-key ca --pem --in /tmp/certs/instance.yml --out /tmp/certs/certs.zip
Expected results

Unzip the certs
sudo unzip /tmp/certs/certs.zip -d /tmp/certs/
Here we set a dir for ease of use
Make the dir for Elasticsearch and Kibana
sudo mkdir /etc/elasticsearch/certs/ /etc/kibana/certs/
copy the certs into place
! Rename "certs/elasticsearch and certs/kibana" to whatever you named your certs when you made them if you are using a different name !
sudo cp -r /tmp/certs/ca/ca.crt /tmp/certs/elasticsearch/* /etc/elasticsearch/certs
sudo cp -r /tmp/certs/ca/ca.crt /tmp/certs/kibana/* /etc/kibana/certs
Backup the existing certs to root home dir (or wherever else you'd like, you might need the CA files sooner or later)
sudo cp -r /tmp/certs/* /root/
Configure Elasticsearch
Backup existing file
sudo mv /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak
Paste new config into the .yml config file
! Rename "elasticsearch" in the http and ssl section to whatever you named your certs when you made them !
sudo bash -c 'cat > /etc/elasticsearch/elasticsearch.yml' << EOF
# ======================== Elasticsearch Configuration =========================
#
# ----------------------------------- Paths ------------------------------------
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
# ---------------------------------- Network -----------------------------------
network.host: 0.0.0.0
http.port: 9200
# --------------------------------- Discovery ----------------------------------
discovery.type: single-node
# ---------------------------------- Various -----------------------------------
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
EOF
Add firewall rules (N.B. Not needed if you are using the Vagrant Centos VM)
Add the port and make sure the ssh service is in the list so you don't lose connectivity
sudo firewall-cmd –permanent --add-port=9200/tcp
sudo firewall-cmd –permanent --add-port=5601/tcp
Now we test to see if Elasticsearch will start
sudo systemctl start elasticsearch
If the services fails to start make sure the certificates are ok and in the right place, if you get a service timeout you will need to edit the systemd service file.
Check the certificates
sudo openssl verify -CAfile /etc/elasticsearch/certs/ca.crt /etc/elasticsearch/certs/elasticsearch.crt
should report OK, if they aren’t found move them from where we made them in /tmp/
Increase the service start timeout
sudo sed -i 's/TimeoutStartSec=75/TimeoutStartSec=500/g' /lib/systemd/system/elasticsearch.service
Validate Elasticsearch
Now that it’s running we can issue a cURL command to get some info
curl -k -XGET https://10.0.2.15:9200/_cat/nodes?pretty
Expected result is

This is a good thing as now that we have setup the security settings Elasticsearch expects us to authenticate to it. We will now generate the required users.
Generate the users
This step uses the internal default Elastic user. You don’t have access to this users password and need to generate the users with the following step.
sudo bash -c '/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto -b > /root/ESUsers.txt'
Check that the certs have been made
sudo cat /root/ESUsers.txt
Expected result

Test that you can now authenticate
curl --cacert /tmp/certs/ca/ca.crt -u elastic:$(sudo grep "elastic =" /root/ESUsers.txt | awk '{print $4}') -k -XGET https://10.0.2.15:9200/_cat/nodes?pretty
Expected result

The reason I am greping the password from the /root/ESUsers.txt file is two fold, firstly it’s very easy to automate now, secondly it doesn’t pollute your .bash_history with a bunch of plaintext passwords.
! The ESUsers.txt file should be copied down to your personal password store and deleted afterwards !
Configure Kibana
Add a new config file
! Change the “server.ssl.certificate” cert name if you used your hostname !
! In this example we are using the password stored in plain text in the kibana.yml file which is a no go for Prod !
! Use the Kibana keystore like it is done in the bootstrap.sh file !
sudo bash -c 'cat > /etc/kibana/kibana.yml' << EOF
# =========================== Kibana Configuration ============================
# -------------------------------- Network ------------------------------------
server.host: 0.0.0.0
server.port: 5601
# ------------------------------ Elasticsearch --------------------------------
elasticsearch.hosts: ["https://10.0.2.15:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "$(sudo grep "kibana_system =" /root/ESUsers.txt | awk '{print $4}')"
# ---------------------------------- Various -----------------------------------
server.ssl.enabled: true
server.ssl.certificate: "/etc/kibana/certs/kibana.crt"
server.ssl.key: "/etc/kibana/certs/kibana.key"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.crt" ]
# ---------------------------------- X-Pack ------------------------------------
xpack.security.encryptionKey: "$(tr -dc A-Za-z0-9 <∕dev∕urandom | head -c 32 ; echo '')"
xpack.encryptedSavedObjects.encryptionKey: "$(tr -dc A-Za-z0-9 <∕dev∕urandom | head -c 32 ; echo '')"
xpack.reporting.encryptionKey: "$(tr -dc A-Za-z0-9 <∕dev∕urandom | head -c 32 ; echo '')"
Don’t worry about the X-Pack settings now they will come into play in a later blog post.
The Vagrant bootstrap.sh script uses the Kibana keystore, this is beyond the scope of this post.
Validate the config
sudo cat /etc/kibana/kibana.yml
Expected result

Start Kibana
sudo systemctl start kibana
Validate it all comes together by visiting https://127.0.0.1:5601 in your favourite browser on your host machine (Not the headless Vagrant VM, but the machine you ran Vagrant from)
You should see the login page like below (After you added an exception for the Self-Signed cert)

To get the elastic users password input this command on the Vagrant VM
sudo grep "elastic =" /root/ESUsers.txt | awk '{print $4}'
If you have made a mistake and do not get the login page at this point you can issue a “vagrant destroy” command and start from the beginning and make sure you copied the Vagrantfile correctly if you followed along with the Vagrant section and have the configs copied correctly.
vagrant.exe destroy
vagrant destroy
Congratulations
Congratulations you now have a demo Elasticsearch + Kibana cluster, in the next blog post we will start to add data. In the mean time have a look around Kibana and it’s features.
You have now built a single node Elasticsearch + Kibana cluster. It won’t do much for you at the moment but watch this space for the next blog posts to see how we can make it work for us.
! Don’t forget this Vagrant setup is not for production !
! Once you have noted all the user/passes down from /root/ESUsers.txt DELETE IT !
! If a nefarious agent gets your Production clusters Elastic password it is GAME OVER !

Automation
If you would like to bring this single node cluster up automatically with all the configs already done please see my GitHub page
“WhY IsN’t It AnSiBlE?!?” The bootstrap.sh provision script is in bash for ease of use and understanding, I would like to make using Elastic as easy as possible. In a later guide we will deploy it with Ansible for “FULL AUTOMATION!”
The readme explains how to bring the VM up with Vagrant
Troubleshooting
Not enough RAM
In the Vagrantfile change this value
From
vb.memory = "8192"
to
vb.memory = "6144”
Once the change is made and the file saved reload the VMs with
vagrant.exe reload
vagrant reload
Not enough vCPU cores
You can reduce the CPU count for the Elastic VM in the Vagrantfile to 2 vCPUs (This might slowdown Elasticsearch and Kibana)
From
vb.cpus = "4"
to
vb.cpus = "2"
Once the change is made and the file saved reload the VMs with
vagrant.exe reload
vagrant reload
Links
SSL creation https://www.golinuxcloud.com/enable-https-ssl-secure-elasticsearch-linux/(very useful)
SSL creation https://www.elastic.co/guide/en/elasticsearch/reference/7.14/security-basic-setup-https.html#encrypt-kibana-elasticsearch(Kind of useful)
Nodes https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html(Useful)
Why we add the X-Pack settings in the Kibana Config https://www.elastic.co/guide/en/kibana/7.15/xpack-security-secure-saved-objects.html(Useful)