Being curious

01 Mar 2018

Installing Filebeat on Raspberry PI 3

How to setup elastic Filebeat from scratch on a Raspberry Pi. At time of writing Elastic do not provide ARM builds for any ELK stack component – so some extra work is required to get this up and going

How to setup elastic Filebeat from scratch on a Raspberry Pi. At time of writing elastic.co do not provide ARM builds for any ELK stack component – so some extra work is required to get this up and going. Guide assumes the following:

  • A working recent instance of the ELK stack is already setup & working
  • A working recent up to date installation of a Debian based operating system using systemd on a later model Raspberry PI
  • Filebeat will be run as the root user

At time of writing this guide: version of the ELK stack used is 6.1.1 & target system was a Raspberry PI 3 running Raspbian GNU Linux 9. If any of the above assumptions don’t apply, what’s listed below will need to be customised accordingly.

Initial Dependencies

Main additional packages required are git, python-pip and virtualenv. Git and python-pip can be installed using the regular package manager:

user@raspberrypi:~ $ sudo apt-get install python-pip git

Pip can then be used to install virtualenv:

user@raspberrypi:~ $ sudo pip install virtualenv

Installing Go

The beats component of ELK is written using go. The default go-1.7.x package that came with Raspbian kept throwing errors during build. At time of writing there wasn’t a more up to date package available, so the current version available from the main website was used.

Raspberry Pi is based on the ARMv6 architecture. Download the current ARMv6 stable release from the main go download site – as of writing the guide the current version is 1.9.2.

user@raspberrypi:~ $ wget https://redirector.gvt1.com/edgedl/go/go1.9.2.linux-armv6l.tar.gz
user@raspberrypi:~ $ sudo tar -C /usr/local -xzf go1.9.2.linux-armv6l.tar.gz
user@raspberrypi:~ $ export PATH=$PATH:/usr/local/go/bin
user@raspberrypi:~ $ go version
go version go1.9.2 linux/arm

It is worthwhile adding PATH=$PATH:/usr/local/go/bin to the ~/.profile or ~/.bash_profile of the account being used. This ensures the account can continue to use go after the next login.

Downloading & Building filebeat

First up a directory location needs to be setup to build filebeat under. For the purpose of this guide the go directory under the users home directory is used. The GOPATH environment variable needs to be set to whatever is chosen for the build process to work.

user@raspberrypi:~ $ export GOPATH=$HOME/go
user@raspberrypi:~ $ mkdir go
user@raspberrypi:~ $ mkdir -p ${GOPATH}/src/github.com/elastic
user@raspberrypi:~ $ cd ${GOPATH}/src/github.com/elastic
user@raspberrypi:~/go/src/github.com/elastic $

Now onto building.

user@raspberrypi:~/go/src/github.com/elastic $ git clone https://github.com/elastic/beats.git
user@raspberrypi:~/go/src/github.com/elastic $ cd beats/
user@raspberrypi:~/go/src/github.com/elastic/beats $ git checkout 23b9e27
user@raspberrypi:~/go/src/github.com/elastic/beats $ cd filebeat/
user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ make
go build -i
user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ make update
New python executable in /home/user/go/src/github.com/elastic/beats/filebeat/build/python-env/bin/python
Installing setuptools, pip, wheel…done.
find: warning: you have specified the -maxdepth option after a non-option argument -type, but options are not positional (-maxdepth affects tests specified before it as well as those specified after it). Please specify options before other arguments.

find: warning: you have specified the -mindepth option after a non-option argument -type, but options are not positional (-mindepth affects tests specified before it as well as those specified after it). Please specify options before other arguments.

Updating generated files for filebeat
make[1]: Entering directory ‘/home/user/go/src/github.com/elastic/beats/libbeat’
make[1]: Leaving directory ‘/home/user/go/src/github.com/elastic/beats/libbeat’
— The index pattern was created under /home/user/go/src/github.com/elastic/beats/filebeat/_meta/kibana/5.x/index-pattern/filebeat.json
— The index pattern was created under /home/user/go/src/github.com/elastic/beats/filebeat/_meta/kibana/default/index-pattern/filebeat.json

Two items to bear in mind as part of building filebeat:

  • The git checkout command is important. Each version of the beats plugin is designed to work with the same version of Logstash and Elasticsearch. The appropriate hashcode for the target release can be found at the elastic/beats release site. When running filbeat for the first time (below) ensure the built version is what was expected.
  • Compiling things with go is supposed to take a large amount of memory. Some of the guides mentioned the need to increase the default amount of swap file available. When completing the above steps on a Raspberry Pi 3 this was not required.

Assuming it all goes well you should now have a new executable named filebeat. It can now be run using the -e (send output to the console) & -v (log info level data) command line flags to confirm it works okay.

user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ ./filebeat -e -v
2018/01/01 08:39:40.336038 beat.go:436: INFO Home path: [/home/user/go/src/github.com/elastic/beats/filebeat] Config path: [/home/user/go/src/github.com/elastic/beats/filebeat] Data path: [/home/user/go/src/github.com/elastic/beats/filebeat/data] Logs path: [/home/user/go/src/github.com/elastic/beats/filebeat/logs]
2018/01/01 08:39:40.336444 beat.go:443: INFO Beat UUID: f87f7ab8-2569-4817-8652-818f0ea26730
2018/01/01 08:39:40.336541 beat.go:203: INFO Setup Beat: filebeat; Version: 6.1.1

Completing the installation

Now that the application is built, it needs to be installed for use. No handy script was found to do this, so the next steps were done manually. First up is reviewing the application install locations. For the purpose of this guide the defaults are used. If needed, feel free to customise.

Type Description Config Option Default Location Debian Default Path
home Home of the Filebeat installation. path.home /usr/share/filebeat
bin The location for the binary files. {path.home}/bin /usr/share/filebeat/bin
config The location for configuration files. path.config {path.home} /etc/filebeat
data The location for persistent data files. path.data {path.home}/data /var/lib/filebeat
logs The location for the logs created by Filebeat. path.log {path.home}/logs /var/log/filebeat

From there there desired paths need to be created and permissions modified:

user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo mkdir /usr/share/filebeat /usr/share/filebeat/bin /etc/filebeat /var/log/filebeat /var/lib/filebeat
user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo mv filebeat /usr/share/filebeat/bin
user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo mv module /usr/share/filebeat/
user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo mv modules.d/ /etc/filebeat/
user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo cp filebeat.yml /etc/filebeat/
user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo chmod 750 /var/log/filebeat
user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo chmod 750 /etc/filebeat/
user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo chown -R root:root /usr/share/filebeat/*

As a final step an initialisation script needs to be placed to support start-up at boot time & starting / stopping / restarting the filebeat service.

user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo vi /lib/systemd/system/filebeat.service

Then paste the below script in.

[Unit]
Description=filebeat
Documentation=https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Wants=userwork-online.target
After=network-online.target

[Service]
ExecStart=/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
Restart=always

[Install]
WantedBy=multi-user.target

Any now to finish it up:

user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo systemctl enable filebeat.service
Created symlink /etc/systemd/system/multi-user.target.wants/filebeat.service → /lib/systemd/system/filebeat.service.
user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo service filebeat start
user@raspberrypi:~/go/src/github.com/elastic/beats/filebeat $ sudo service filebeat status
● filebeat.service – filebeat
Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2018-01-01 20:35:33 AEDT; 4s ago
Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Main PID: 6481 (filebeat)
CGroup: /system.slice/filebeat.service
└─6481 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.conf

Jan 01 20:35:33 raspberrypi systemd[1]: Started filebeat.

Filebeat configuration can now be set in /etc/filebeat/filebeat.yml and the service started, stopped & restarted using normal system commands.

The following links were used as part of working my way through this guide: