Intro
Preamble
If you are new to Elastic and would like to know how to configure it read the blog post here
If you are new to Elasticsearch please read the blog post entitled “What is Elastic and Why Should You Care - TBD” it goes into further detail regarding what Elastic is, and how we will use it.
In this tutorial we will bring up a single node cluster for demonstration purposes only, the setup we are doing now forms preparatory work to be able to test more advanced features of Elastic in later blog posts.
Don’t forget this is not intended for Production. Only as reference material.
If you don’t understand some of the terms on in this tutorial please see if it’s hyperlinked or in the blog post about Elasticsearch or send me an email.
Prerequisites
Required Software
- An Elastic cluster with Elasticsearch and Kibana configured for secure settings (Config to follow)
- OS (Centos 7 if you plan on running Elasticsearch on a VM be sure to add at least 8GB of RAM and 2 vCPU cores)
- VirtualBox
- Vagrant (The Vagrant provisioning scripts we will use are based on VirtualBox)
- Elasticsearch 8.0.0
- Kibana 8.0.0
- Vi/Vim/Emacs (or your favourite editing program)
- Browser (Firefox, Chrome, etc)
- Between 8-16 GB RAM (The ES cluster we will bring up uses at least 8GB) See Troubleshooting for advice
- More than 2 CPU cores that have Hyperthreading (if you press Crtl + Shift + Esc keys together and go to the Task manager -> performance -> cpu tab count the tiles if it’s above 4 you’re ok) See Troubleshooting for advice
Conventions used in this blog post
Code will appear as code
Things to be cautious around are wrapped like this ! Warning ! Be careful and take heed.
Terminals are presented like this:
Windows PowerShell
example
Linux
Root user
example
Unprivileged user
example
There is a small copy button inside the code blocks it looks like this when moused over (It won’t copy hostname information so you can paste with ease)

Lab Diagram

Building the Vagrant VM
This can be skipped if you have your own cluster already built, however you might have issues around the Fleet server certificates.
OS
Windows
Provision Centos 7 VM
To provision the VM once VirtualBox and Vagrant have been installed run this command in PowerShell (Replace USERNAME with your username)
The code can be found on my Github here
Replace USERNAME with your username
mkdir -p C:\Users\USERNAME\Boxes\Elastic_8.0_Fleet
Next cd to that DIR
cd C:\Users\USERNAME\Boxes\Elastic_8.0_Fleet
Download the Vagrantfile and ESBootstrap.sh files from Github here to the dir you just created.
What the above Vagrantfile configures is 2 VMs, one for Elasticsearch and one for our Elastic Agent (This can be collapsed into 1 VM but for this tutorial we are using two to explain some of the Fleet server and Agent concepts).
We also have configured 2 private Nics so the VMs can more easily communicate with each other and we can expand this base into a more advanced lab later.
The ESBootstrap.sh
script will provision a single node cluster in an automated and secure way.
Bring the Boxes up
vagrant.exe up
Log into the Elastic box to validate that everything worked thus far
If you get an error message like Cannot find a valid baseurl for repo: base/7/x86_64
destroy the VMs with the vagrant destroy
command and re-provision them.
vagrant.exe ssh Elastic
You can see that the terminal curser changes to a Linux terminal which means the box is up and running successfully.
Verify that Elasticsearch and Kibana have installed correctly while logged into the Elastic machine
sudo head -n 3 /etc/elasticsearch/elasticsearch.yml
sudo head -n 3 /etc/kibana/kibana.yml
If the above commands output text then the installation worked
Congrats you just built a secure Elasticsearch + Kibana cluster in 1 command, lucky you.
Linux
Provision Centos 7 VM
To provision the VM once VirtualBox and Vagrant have been installed run this command in BASH (Replace USERNAME with your username)
The code can be found on my Github here
Replace USERNAME with your username
mkdir -p /home/USERNAME/Boxes/Elastic_8.0_Fleet
Next cd to that DIR
cd !$
Download the Vagrantfile and ESBootstrap.sh files from Github here to the dir you just created.
What the above Vagrantfile configures is 2 VMs, one for Elasticsearch and one for our Elastic Agent (This can be collapsed into 1 VM but for this tutorial we are using two to explain some of the Fleet server and Agent concepts).
We also have configured 2 private Nics so the VMs can more easily communicate with each other and we can expand this base into a more advanced lab later.
The ESBootstrap.sh
script will provision a single node cluster in an automated and secure way.
Bring the Boxes up
vagrant up
Log into the Elastic box to validate that everything worked thus far
If you get an error message like Cannot find a valid baseurl for repo: base/7/x86_64
destroy the VMs with the vagrant destroy
command and re-provision them.
vagrant ssh Elastic
You can see that the terminal curser changes to a Linux terminal which means the box is up and running successfully.
Verify that Elasticsearch and Kibana have installed correctly while logged into the Elastic machine
sudo head -n 3 /etc/elasticsearch/elasticsearch.yml
sudo head -n 3 /etc/kibana/kibana.yml
If the above commands output text then the installation worked
Congrats you just built a secure Elasticsearch + Kibana cluster in 1 command, lucky you.
Configure Fleet Server
You will notice some new files in the directory you ran Vagrant from (Under the hood Vagrant provisions the VMs with a shared folder with the host system and mounts it under /vagrant)
The bootstrap.sh script downloaded Elasticsearch, Kibana and Elastic-Agent and installed ES+Kib from their downloaded packages (We will install an Elastic Agent manually to get a feel for how we can automate it later). If you run vagrant destroy
now you will lose the VM and secure configs, but not the downloaded packages so we don’t spam Elastic package repos. If you run vagrant up –provision
again you won’t download new packages but it will install and configure ES+Kib like new.
Connect to Kibana
Open a web-browser on your host system (The system that you ran Vagrant from) and go to https://10.0.0.10:5601 and ignore the certificate warning as these are self-signed certs
The Elastic user password will have been printed to the console that you ran vagrant up
from in the section "-- Security autoconfiguration information --"
! Do not leave the Elastic user’s password laying around anywhere in production !
Copy and past the results into the password prompt and the username is “elastic”

Navigate to the Fleet settings section (It will take some time to load as this is the first time the system has run so it does some background housekeeping / config)

Configure this section for a Fleet server (a Fleet server is an Elastic Agent running a special module)
Step 1 Download
Download the Agent (Skip this if you have spun up the Elastic+Fleet Vagrant VM as it’s already download for you)
Decompress it like so (we output to the /opt dir for ease of access/use)
sudo tar -xvf /vagrant/elastic-agent-8.0.0-linux-x86_64.tar.gz -C /opt/
https://www.elastic.co/downloads/elastic-agent
Step 2 the Agent Policy
Select the “Default Fleet Server” one as that is what we want to setup
Step 3 the deployment mode
Select Production as this provide the most secure option
Step 4 add the Fleet server and Elasticsearch server
Open the Fleet settings and add the below settings
Fleet Server host https://10.0.0.10:8220 and click add host
Then click fleet settings

Click the edit button

Set the Elasticsearch Host (The IP:Port that agents will output to)
Add ssl.certificate_authorities: ["/vagrant/ca.crt"]
to the Advanced YAML configuration section
If you are following along at home with the Elastic+Fleet Vagrant VM these are the settings you’ll need (N.B. they are all HTTPS this is needed if you set it to HTTP it will fail (To verify and T-Shoot problems on the Elastic VM run tail -f /var/log/elasticsearch/elasticsearch.log
and if you see java error messages saying "connection reset due to sending http traffic to a https port" this is the cause, more trouble shooting is at the end of the blog))
This is partly why we publish the CA cert into the shared /vagrant dir.

Save and apply the settings and go back to the Agents page
Step 5 generate a service token
Click the Generate button
Step 6 Start the Fleet server
It’s all coming together now. Run the following command on the Elastic VM (Replace <Token> and <Policy> with what you get in Step 6)in the GUI
We’ve cheated a bit and generated the certificates in the bootstrap.sh script to make this as easy as possible
If you are trying to integrate Fleet into an existing ES cluster please follow this guide to generate Fleet server certificates from your existing certificates (if you have any)
If you are using the Elastic SaaS or ECE/ECK things are a bit different and “This is not the guide you are looking for”
Copy the below into a text editor first to add the "Token" and "Policy" before pasting into the terminal
sudo /opt/elastic-agent-8.0.0-linux-x86_64/elastic-agent install --url=https://10.0.0.10:8220 \
--fleet-server-es=https://10.0.0.10:9200 \
--fleet-server-service-token=<Token> \
--fleet-server-policy=<Policy> \
--certificate-authorities=/vagrant/ca.crt \
--fleet-server-es-ca=/etc/pki/fleet/ca.crt \
--fleet-server-cert=/etc/pki/fleet/fleet.crt \
--fleet-server-cert-key=/etc/pki/fleet/fleet.key
Expected Result in terminal
sudo /opt/elastic-agent-8.0.0-linux-x86_64/elastic-agent install
...
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]y
...
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.
In Webpage

This agent named “elastic-8” is now a Fleet server that future Elastic Agents can connect to
To verify that it’s worked (The output in Kibana can be misleading as the Elastic Agent has to successfully communicate with two distinct instances; the first is Kibana as seen above, the second is Elasticsearch which isn’t shown above so navigate to the agent detail page by clicking on it’s Host name)
Now view the logs page
If you don’t get output in the below page something has gone wrong (most likely somewhere in the SSL settings so if you are deploying on your own infra spin up the Vagrant VM and copy settings from the bootstrap.sh. If problems persist consult the T-Shoot section or send me an email or open an issue on Github)

Install Elastic Agent on a Different Server
It’s all well and good to have Elastic Agent (The Fleet Server) on one machine but that’s no fun.
Lets fix that by adding the Elastic Agent to the Agent VM, “double your VMs double your fun”
Log into the Agent VM once you’ve exited out of the Elastic VM
vagrant ssh Agent
In the web-browser
Install this Agent like so clicking the “Add Agent” button in the Webpage will show you how to get these values

Step 1 Download the Agent
Download the Agent (Skip this if you have spun up the Elastic+Fleet Vagrant VM as it’s already download for you)
Decompress the Agent like before
sudo tar -xvf /vagrant/elastic-agent-8.0.0-linux-x86_64.tar.gz -C /opt/
Step 2 Select the policy
We will use the Default policy that comes pre packaged for now (But you can add policies as you need which is outside the scope of this guide)
Step 3 Enroll the Agent
Copy and paste what is in the box and add the CA cert so the command will look like the below
sudo /opt/elastic-agent-8.0.0-linux-x86_64/elastic-agent install -f \
--url=https://10.0.0.10:8220 \
--enrollment-token=<Token> \
--certificate-authorities=/vagrant/ca.crt
Verify the output (it may take some time no more than 5 min)
Look under this new Agent-8 Log tab to verify everything has worked ok so far

Logs (as before this Agent needs to successfully connect to two distinct places, the Fleet server and Elasticsearch)

This verifies that it’s enrolled in Fleet, seen in Kibana, and outputting data to Elasticsearch (if you don’t see any data here same as before there is an issue with how the Agent is writing data to Elasticsearch, consult the T-Shooting guide below)
You can explore some of the default data in the Observability section

N.B. you will only see one VM here (The Fleet server doesn’t have a system module policy)
Congratulations
Congratulations you now have a base Fleet server and Elastic Agent enrolled in it sending data to Elasticsearch to be viewed at your leisure in Kibana
Troubleshooting
Not enough RAM
If you don’t have enough RAM for both VMs to run successfully you can try reduce the Elastic VM down to 6GB (6144 Mb), this is very much not recommended however but might work in a pinch. Elasticsearch advise at least 8GB RAM as the JVM heap is RAM hungry.
In the Vagrantfile change this value
From
v.customize ["modifyvm", :id, "--memory", 8192]
to
v.customize ["modifyvm", :id, "--memory", 6144]
Once the change is made and the file saved reload the VMs with
vagrant reload
I’m not sure how the agent VM will handle less RAM but you could also try reducing it’s RAM from 1024 Mb down to 512 Mb using the same process as above.
Not enough vCPU cores
You can reduce the CPU count for the Elastic VM in the Vagrantfile to 2 vCPUs (This will slowdown Elasticsearch and Kibana quite a bit)
From
v.customize ["modifyvm", :id, "--cpus", 4]
to
v.customize ["modifyvm", :id, "--cpus", 2]
Once the change is made and the file saved reload the VMs with
vagrant reload
Uninstall Agent
Uninstall the agent with
sudo /usr/bin/elastic-agent uninstall
Links
Install the Elastic Agent https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html (Very useful)
Encrypt traffic in a self-managed cluster https://www.elastic.co/guide/en/fleet/current/secure-connections.html (Very useful)
Fleet Server Spec https://www.elastic.co/guide/en/fleet/current/fleet-server.html (Very Useful Diagrams)
Fleet Overview https://www.elastic.co/guide/en/fleet/current/fleet-overview.html (Very Useful Explanations)
Download the Elastic Agent https://www.elastic.co/downloads/elastic-agent (Needed ofc)
[discuss][Fleet] Custom CAs and SSL options https://github.com/elastic/kibana/issues/72718 (Useful for me to work out some of the issues)
Elastic Agent Troubleshooting guide https://www.elastic.co/guide/en/fleet/current/fleet-troubleshooting.html (Useful for T-Shooting common problems)
Elasticsearch JVM https://www.elastic.co/blog/a-heap-of-trouble (Good info re RAM sizing)
Fleet TLS settings https://www.elastic.co/guide/en/fleet/8.1/secure-connections.html (Good info and guide for TLS)