How to install Fleet and add Elastic Agent Data to Elasticsearch 8.0

Intro

Preamble

If you are new to Elastic and would like to know how to configure it read the blog post here

If you are new to Elasticsearch please read the blog post entitled “What is Elastic and Why Should You Care - TBD” it goes into further detail regarding what Elastic is, and how we will use it.

In this tutorial we will bring up a single node cluster for demonstration purposes only, the setup we are doing now forms preparatory work to be able to test more advanced features of Elastic in later blog posts.

Don’t forget this is not intended for Production. Only as reference material.

If you don’t understand some of the terms on in this tutorial please see if it’s hyperlinked or in the blog post about Elasticsearch or send me an email.

Prerequisites

Required Software

• An Elastic cluster with Elasticsearch and Kibana configured for secure settings (Config to follow)
• OS (Centos 7 if you plan on running Elasticsearch on a VM be sure to add at least 8GB of RAM and 2 vCPU cores)
• VirtualBox
• Vagrant (The Vagrant provisioning scripts we will use are based on VirtualBox)
• Elasticsearch 8.0.0
• Kibana 8.0.0
• Vi/Vim/Emacs (or your favourite editing program)
• Browser (Firefox, Chrome, etc)
• Between 8-16 GB RAM (The ES cluster we will bring up uses at least 8GB) See Troubleshooting for advice
• More than 2 CPU cores that have Hyperthreading (if you press Crtl + Shift + Esc keys together and go to the Task manager -> performance -> cpu tab count the tiles if it’s above 4 you’re ok) See Troubleshooting for advice

Conventions used in this blog post

Code will appear as code
Things to be cautious around are wrapped like this ! Warning ! Be careful and take heed.

Terminals are presented like this:
Windows PowerShell

example

Linux
Root user

example

Unprivileged user

example

There is a small copy button inside the code blocks it looks like this when moused over (It won’t copy hostname information so you can paste with ease)

Lab Diagram

Building the Vagrant VM

This can be skipped if you have your own cluster already built, however you might have issues around the Fleet server certificates.

OS

mkdir -p C:\Users\USERNAME\Boxes\Elastic_8.0_Fleet

cd C:\Users\USERNAME\Boxes\Elastic_8.0_Fleet

vagrant.exe up

vagrant.exe ssh Elastic

sudo head -n 3 /etc/elasticsearch/elasticsearch.yml

sudo head -n 3 /etc/kibana/kibana.yml

mkdir -p /home/USERNAME/Boxes/Elastic_8.0_Fleet

cd !$

vagrant up

vagrant ssh Elastic

sudo head -n 3 /etc/elasticsearch/elasticsearch.yml

sudo head -n 3 /etc/kibana/kibana.yml

Configure Fleet Server

You will notice some new files in the directory you ran Vagrant from (Under the hood Vagrant provisions the VMs with a shared folder with the host system and mounts it under /vagrant)
The bootstrap.sh script downloaded Elasticsearch, Kibana and Elastic-Agent and installed ES+Kib from their downloaded packages (We will install an Elastic Agent manually to get a feel for how we can automate it later). If you run vagrant destroy now you will lose the VM and secure configs, but not the downloaded packages so we don’t spam Elastic package repos. If you run vagrant up –provision again you won’t download new packages but it will install and configure ES+Kib like new.

Connect to Kibana

Open a web-browser on your host system (The system that you ran Vagrant from) and go to https://10.0.0.10:5601 and ignore the certificate warning as these are self-signed certs

The Elastic user password will have been printed to the console that you ran vagrant up from in the section "-- Security autoconfiguration information --"

! Do not leave the Elastic user’s password laying around anywhere in production !

Copy and past the results into the password prompt and the username is “elastic”

Navigate to the Fleet settings section (It will take some time to load as this is the first time the system has run so it does some background housekeeping / config)

Configure this section for a Fleet server (a Fleet server is an Elastic Agent running a special module)

Step 1 Download

Download the Agent (Skip this if you have spun up the Elastic+Fleet Vagrant VM as it’s already download for you)
Decompress it like so (we output to the /opt dir for ease of access/use)

sudo tar -xvf /vagrant/elastic-agent-8.0.0-linux-x86_64.tar.gz -C /opt/

https://www.elastic.co/downloads/elastic-agent

Step 2 the Agent Policy

Select the “Default Fleet Server” one as that is what we want to setup

Step 3 the deployment mode

Select Production as this provide the most secure option

Step 4 add the Fleet server and Elasticsearch server

Open the Fleet settings and add the below settings
Fleet Server host https://10.0.0.10:8220 and click add host
Then click fleet settings

Click the edit button

Set the Elasticsearch Host (The IP:Port that agents will output to)
Add ssl.certificate_authorities: ["/vagrant/ca.crt"] to the Advanced YAML configuration section
If you are following along at home with the Elastic+Fleet Vagrant VM these are the settings you’ll need (N.B. they are all HTTPS this is needed if you set it to HTTP it will fail (To verify and T-Shoot problems on the Elastic VM run tail -f /var/log/elasticsearch/elasticsearch.log and if you see java error messages saying "connection reset due to sending http traffic to a https port" this is the cause, more trouble shooting is at the end of the blog))
This is partly why we publish the CA cert into the shared /vagrant dir.

Save and apply the settings and go back to the Agents page

Step 5 generate a service token

Click the Generate button

Step 6 Start the Fleet server

It’s all coming together now. Run the following command on the Elastic VM (Replace <Token> and <Policy> with what you get in Step 6)in the GUI
We’ve cheated a bit and generated the certificates in the bootstrap.sh script to make this as easy as possible

If you are trying to integrate Fleet into an existing ES cluster please follow this guide to generate Fleet server certificates from your existing certificates (if you have any)

If you are using the Elastic SaaS or ECE/ECK things are a bit different and “This is not the guide you are looking for”
Copy the below into a text editor first to add the "Token" and "Policy" before pasting into the terminal

sudo /opt/elastic-agent-8.0.0-linux-x86_64/elastic-agent install --url=https://10.0.0.10:8220 \
 --fleet-server-es=https://10.0.0.10:9200 \
 --fleet-server-service-token=<Token> \
 --fleet-server-policy=<Policy> \
 --certificate-authorities=/vagrant/ca.crt \
 --fleet-server-es-ca=/etc/pki/fleet/ca.crt \
 --fleet-server-cert=/etc/pki/fleet/fleet.crt \
 --fleet-server-cert-key=/etc/pki/fleet/fleet.key

Expected Result in terminal

sudo /opt/elastic-agent-8.0.0-linux-x86_64/elastic-agent install
...
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]y
...
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.

In Webpage

This agent named “elastic-8” is now a Fleet server that future Elastic Agents can connect to

To verify that it’s worked (The output in Kibana can be misleading as the Elastic Agent has to successfully communicate with two distinct instances; the first is Kibana as seen above, the second is Elasticsearch which isn’t shown above so navigate to the agent detail page by clicking on it’s Host name)

Now view the logs page

If you don’t get output in the below page something has gone wrong (most likely somewhere in the SSL settings so if you are deploying on your own infra spin up the Vagrant VM and copy settings from the bootstrap.sh. If problems persist consult the T-Shoot section or send me an email or open an issue on Github)

Install Elastic Agent on a Different Server

It’s all well and good to have Elastic Agent (The Fleet Server) on one machine but that’s no fun.
Lets fix that by adding the Elastic Agent to the Agent VM, “double your VMs double your fun”

Log into the Agent VM once you’ve exited out of the Elastic VM

vagrant ssh Agent

In the web-browser

Install this Agent like so clicking the “Add Agent” button in the Webpage will show you how to get these values

Step 1 Download the Agent

Download the Agent (Skip this if you have spun up the Elastic+Fleet Vagrant VM as it’s already download for you)
Decompress the Agent like before

sudo tar -xvf /vagrant/elastic-agent-8.0.0-linux-x86_64.tar.gz -C /opt/

Step 2 Select the policy

We will use the Default policy that comes pre packaged for now (But you can add policies as you need which is outside the scope of this guide)

Step 3 Enroll the Agent

Copy and paste what is in the box and add the CA cert so the command will look like the below

sudo /opt/elastic-agent-8.0.0-linux-x86_64/elastic-agent install -f \
  --url=https://10.0.0.10:8220 \
  --enrollment-token=<Token> \
  --certificate-authorities=/vagrant/ca.crt

Verify the output (it may take some time no more than 5 min)

Look under this new Agent-8 Log tab to verify everything has worked ok so far

Logs (as before this Agent needs to successfully connect to two distinct places, the Fleet server and Elasticsearch)

This verifies that it’s enrolled in Fleet, seen in Kibana, and outputting data to Elasticsearch (if you don’t see any data here same as before there is an issue with how the Agent is writing data to Elasticsearch, consult the T-Shooting guide below)

You can explore some of the default data in the Observability section

N.B. you will only see one VM here (The Fleet server doesn’t have a system module policy)

Congratulations

Congratulations you now have a base Fleet server and Elastic Agent enrolled in it sending data to Elasticsearch to be viewed at your leisure in Kibana

Troubleshooting

Not enough RAM

If you don’t have enough RAM for both VMs to run successfully you can try reduce the Elastic VM down to 6GB (6144 Mb), this is very much not recommended however but might work in a pinch. Elasticsearch advise at least 8GB RAM as the JVM heap is RAM hungry.

In the Vagrantfile change this value

From
v.customize ["modifyvm", :id, "--memory", 8192]
to
v.customize ["modifyvm", :id, "--memory", 6144]

Once the change is made and the file saved reload the VMs with

vagrant reload

I’m not sure how the agent VM will handle less RAM but you could also try reducing it’s RAM from 1024 Mb down to 512 Mb using the same process as above.

Not enough vCPU cores

You can reduce the CPU count for the Elastic VM in the Vagrantfile to 2 vCPUs (This will slowdown Elasticsearch and Kibana quite a bit)

From
v.customize ["modifyvm", :id, "--cpus", 4]
to
v.customize ["modifyvm", :id, "--cpus", 2]

Once the change is made and the file saved reload the VMs with

vagrant reload

Uninstall Agent

Uninstall the agent with

sudo /usr/bin/elastic-agent uninstall

Links

Install the Elastic Agent https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html (Very useful)

Encrypt traffic in a self-managed cluster https://www.elastic.co/guide/en/fleet/current/secure-connections.html (Very useful)

Fleet Server Spec https://www.elastic.co/guide/en/fleet/current/fleet-server.html (Very Useful Diagrams)

Fleet Overview https://www.elastic.co/guide/en/fleet/current/fleet-overview.html (Very Useful Explanations)

Download the Elastic Agent https://www.elastic.co/downloads/elastic-agent (Needed ofc)

[discuss][Fleet] Custom CAs and SSL options https://github.com/elastic/kibana/issues/72718 (Useful for me to work out some of the issues)

Elastic Agent Troubleshooting guide https://www.elastic.co/guide/en/fleet/current/fleet-troubleshooting.html (Useful for T-Shooting common problems)

Elasticsearch JVM https://www.elastic.co/blog/a-heap-of-trouble (Good info re RAM sizing)

Fleet TLS settings https://www.elastic.co/guide/en/fleet/8.1/secure-connections.html (Good info and guide for TLS)