We sometimes write.

Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our achievements and share some solutions.

Logstash to Logstash log forwarding using the Lumberjack output plugin

Published on May 28th 2020 - see original post


In a usual ELK setup, one would run application servers logging to a central Logstash server and this Logstash writes log events into an Elasticsearch index.

But sometimes it might be required to forward logs from a central Logstash to another Logstash server. A practical example: Your applications run on different premises or cloud providers but you still want to have all the logs in a central place. For such a scenario a "local" Logstash server, which is used as a forwarder to the central Logstash server, can be used. The question is: Which output plugin should be used to forward the logs?

Once upon a time: logstash-forwarder

A couple of years ago, the logstash-forwarder project was created to solve exactly this: Pick up logs and forward them to one or more Logstash servers "listening for our messages".

However logstash-forwarder was replaced by Filebeat:

The filebeat project replaces logstash-forwarder. Please use that instead.

Interestingly this project was renamed from "lumberjack" to "logstash-forwarder" as can be read in the README:

This project was recently renamed from 'lumberjack' to 'logstash-forwarder' to make its intended use clear. The 'lumberjack' name now remains as the network protocol, and 'logstash-forwarder' is the name of the program.

We'll come back to the lumberjack name later again...

Filebeat output?

So if Filebeat should be used instead, is there an actual logstash-output-filebeat plugin available? No, there isn't (as of this writing). The idea behind using Filebeat as a forwarder to another Logstash server would be for the local Logstash to write local logs into files. These files are then picked up by Filebeat and sent to the central Logstash server.

Sounds messy? It is. Not only will you lose time by writing disks to log (by Logstash) and reading them again (by Filebeat), it also means to install a second daemon (Filebeat) which uses resources on that local Logstash server. 

It would surely work, but do we really want to involve another daemon? Nope.

Finding the right Logstash output plugin

The challenge now is to find the right Logstash output plugin. Sure, there are the basic tcp and udp output plugins, which simply connect to a Logstash listener on the given port - but is that the right choice?

Going through the list of Logstash output plugins, one name catches the eye: lumberjack.

Wait, wasn't lumberjack the original name of the logstash-forwarder? A quick read in the lumberjack output plugin documentation doesn't really show a use case for this output, but it's certainly worth a try.

Using lumberjack as log forwarder

The lumberjack output basically uses the lumberjack protocol, which was picked up and replaced by Filebeat (as mentioned above). This means: Our local Logstash can use lumberjack as output and our central Logstash can use the beats input to receive these log events. 

Logstash to Logstash log forwarding using lumberjack

In order to use the lumberjack output, SSL certificate exchange between the local and the central Logstash servers must happen. The ssl_certificate parameter is required, it won't work without it. The SSL key and certificate need to be configured on the central (receiving) Logstash server and configured in the beats intput definition. On the local (sending) Logstash, it's enough to use the same SSL certificate.

To create a very basic SSL key and certificate on the central Logstash server (make sure to set CN to your central Logstash FQDN):

root@central:~# openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.cert -subj /CN=logstash.example.com

This command creates two files:

Move both files into /etc/logstash/ and configure a new beats input with the paths to these files:

# Beats input for collecting logs from other Logstash servers using lumberjack
input {
  beats {
    codec => json
    port => 6000
    ssl => true
    ssl_certificate => "/etc/logstash/lumberjack.cert"
    ssl_key => "/etc/logstash/lumberjack.key"
  }
}

On the sending side, our "local" Logstash, the lumberjack output can be defined, using the same SSL certificate:

output {
    lumberjack {
      codec => json
      hosts => "logstash.example.com"
      ssl_certificate => "/etc/logstash/lumberjack.cert"
      port => 6000
    }
}

Don't forget to install the lumberjack output plugin as it is not part of the default output plugins:

root@local:~# /usr/share/logstash/bin/logstash-plugin install --no-verify logstash-output-lumberjack
Installing logstash-output-lumberjack
Installation successful

Restart Logstash and that's it already. As long as your local Logstash can communicate with the central Logstash with the given tcp port (here tcp/6000), log events should now arrive on the central Logstash server and stored into the Elasticsearch index.