Getting logstash-forwarder working

6 Comments / logstash / By

I’m doing some work with logstash-forwarder as part of a consolidation effort to get all the logs into one place, and using logstash to put them into Elasticsearch.

logstash-fowarder requires the Go programming language, and that will be our first step.

Installing go, the programming language

I am assuming you are installing go at /usr/local; adjust the path to suit.

I am installing from source because older versions of go will not work. The default package for the OS I was using was too old.

The first step in installing logstash-forwarder is installing The Go Programming Language. Here are the steps I followed:

# mkdir /usr/local/
# wget http://golang.org/dl/go1.3.linux-amd64.tar.gz
# tar -xzf go1.3.linux-amd64.tar.gz (this step will create a subdirectory: go)

Adjust your path so you can pick up go without specifying the full path.

Installing logstash-forwarder

To install logstash-fowarder, follow these steps:

# cd /usr/local
# git clone git://github.com/elasticsearch/logstash-forwarder.git
# cd logstash-forwarder
# go build

The logstash-forwarder configuration file

I put the logstash-forwarder configuration file at /usr/local/etc/logstash-forwarder/forwarding.conf and it contains this:

{
  "network": {
    "servers": [ "localhost:5000" ],

    "ssl certificate": "/usr/local/etc/logstash-forwarder/logstash-forwarder.crt",
    "ssl key": "/usr/local/etc/logstash-forwarder/logstash-forwarder.key",
    "ssl ca": "/usr/local/etc/logstash-forwarder/ca.pem",

    "timeout": 15
  },

  "files": [
    {
      "paths": [
        "/home/ubuntu/nginx_data/empty.log"
      ],
      "fields": { "type": "nginx-access" }
    }
  ]
}

In this case, logs will be forwarded to port 5000 on localhost. In the next section, we will next configure logstash to listen on that port for incoming data.

In this case, the file /home/ubuntu/nginx_data/empty.log will be forwarded.

The ssl certificate in question must be set for localhost. In my case, I created my own CA and used that to create my own certificates.

Configuring logstash to listen for logstash-forwarder

This is the entry I added to the configuration file of an existing logstash instance:

input {
  lumberjack {
    # The port to listen on
    port => 5000

    # The paths to your ssl cert and key
    ssl_certificate => "logstash-receiver.crt"
    ssl_key         => "logstash-receiver.key"

    # Set this to whatever you want.
    type => "somelogs"
  }
}

I used the same ssl certificate that I used for logstash-forwarder but the key part: make sure logstash-forwarder can validate this certificate. In this case, it can.

Start logstash-fowarder

Here is how I started logstash-forwarder:

# cd /usr/local/logstash-forwarder/
# ./logstash-forwarder -config /usr/local/etc/logstash-forwarder/forwarding.conf
2014/08/13 13:04:45 publisher init
2014/08/13 13:04:45 {
  "network": {
    "servers": [ "localhost:5000" ],

    "ssl certificate": "/usr/local/etc/logstash-forwarder/logstash-forwarder.crt",
    "ssl key": "/usr/local/etc/logstash-forwarder/logstash-forwarder.key",
    "ssl ca": "/usr/local/etc/logstash-forwarder/ca.pem",

    "timeout": 15
  },

  "files": [
    {
      "paths": [
        "/home/ubuntu/nginx_data/empty.log"
      ],
      "fields": { "type": "nginx-access" }
    }
  ]
}
2014/08/13 13:04:45.279272 Loading registrar data from /usr/local/logstash-forwarder/.logstash-forwarder
2014/08/13 13:04:45.279377 Waiting for 1 prospectors to initialise
2014/08/13 13:04:45.279650 Resuming harvester on a previously harvested file: /home/ubuntu/nginx_data/empty.log
2014/08/13 13:04:45.279719 Registrar will re-save state for /home/ubuntu/nginx_data/empty.log
2014/08/13 13:04:45.279741 All prospectors initialised with 1 states to persist
2014/08/13 13:04:45.279803 Started harvester at position 98 (current offset now 98): /home/ubuntu/nginx_data/empty.log
2014/08/13 13:04:45.279944 Loading client ssl certificate: /usr/local/etc/logstash-forwarder/logstash-forwarder.crt and /usr/local/etc/logstash-forwarder/logstash-forwarder.key
2014/08/13 13:04:45.309917 Setting trusted CA from file: /usr/local/etc/logstash-forwarder/ca.pem
2014/08/13 13:04:45.310837 Connecting to [127.0.0.1]:5000 (localhost)
2014/08/13 13:04:45.310987 Failure connecting to 127.0.0.1: dial tcp 127.0.0.1:5000: connection refused
2014/08/13 13:04:46.311289 Connecting to [127.0.0.1]:5000 (localhost)
2014/08/13 13:04:46.311390 Failure connecting to 127.0.0.1: dial tcp 127.0.0.1:5000: connection refused
2014/08/13 13:04:47.311703 Connecting to [127.0.0.1]:5000 (localhost)
...

That looping will continue until I start logstash.

Errors I encountered

While getting logstash-forwarder running, I saw these errors when I used certificates created with the method mentioned in their documentation. They went away after I created my own certificates for localhost:

2014/08/12 16:25:37.877849 Failed to tls handshake with 127.0.0.1 x509: certificate is valid for , not localhost

In attempt to get around that error, I removed the ssl ca clause from the configuration. I then encountered this error:

2014/08/12 17:13:23.132688 Failed to tls handshake with 127.0.0.1 x509: certificate signed by unknown authority

I then abandoned their recommendations and started my own CA. The resulting certificates failed on my first attempt. This error went away after I created a cert for localhost:

2014/08/12 18:20:30.835630 Failed to tls handshake with 127.0.0.1 x509: certificate is valid for logstash-forwarder.example.org, not localhost

Starting logstash

Here is how I started logstash:

# cd /usr/local/logstash/logstash-1.4.2/
# bin/logstash -f logstash.conf
Using milestone 1 input plugin 'lumberjack'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}

What happened next?

Once I started l started logstash, I saw this in the logstash-forwarder window:

2014/08/13 13:11:59.333029 Failure connecting to 127.0.0.1: dial tcp 127.0.0.1:5000: connection refused
2014/08/13 13:12:00.333348 Connecting to [127.0.0.1]:5000 (localhost)
2014/08/13 13:12:00.480856 Connected to 127.0.0.1
2014/08/13 13:12:00.535690 Registrar received 20 events

This shows logstash-forwarder finally connecting to logstash and sending the 20 lines which was already in that file.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

6 thoughts on “Getting logstash-forwarder working”

  1. MacLing

    i have configure my OWN CA, and test with openssl ==> connected.

    -sh-4.1$ openssl s_client -connect 10.147.252.13:6514 -CAfile LINGGA-cacert.pem -cert client11-cert.pem -key client11privkey.pem 
CONNECTED(00000003)

    but when running with logstash-forwarder :

    2014/09/09 11:40:22.110276 Waiting for 2 prospectors to initialise
2014/09/09 11:40:22.110517 Launching harvester on new file: /var/log/messages
2014/09/09 11:40:22.110717 Failed opening /var/log/messages: open /var/log/messages: permission denied
2014/09/09 11:40:22.110734 All prospectors initialised with 0 states to persist
2014/09/09 11:40:22.111034 Loading client ssl certificate: /var/lib/nagios/clientno/client11-cert.pem and /var/lib/nagios/clientno/client11privkey.pem
2014/09/09 11:40:23.357242 Setting trusted CA from file: /var/lib/nagios/clientno/LINGGA-cacert.pem
2014/09/09 11:40:23.357568 Failed to parse a certificate: /var/lib/nagios/clientno/LINGGA-cacert.pem

    config-file :

    $ more /etc/logstash-forwarder
{
  "network": {
  "servers": [ "10.147.252.13:6514" ],
  "ssl certificate": "/var/lib/nagios/clientno/client11-cert.pem",
  "ssl key": "/var/lib/nagios/clientno/client11privkey.pem",
  "ssl ca": "/var/lib/nagios/clientno/LINGGA-cacert.pem",
  "timeout": 15
},
  "files": [
   {
     "paths": [ "/var/log/messages" ],
     "fields": { "type": "syslog" }
   }, {
     "paths": [ "/var/log/httpd/access_log" ],
     "fields": { "type": "apache" }
   }
   ]
 }

    Please advice what’s the missing step and config…

    Thank you.

  2. Dan Langille

    I suspect the issue is with /var/lib/nagios/clientno/LINGGA-cacert.pem

    logstash is unable to parse a certificate from that file.

    That file should look something like this:

    —–BEGIN CERTIFICATE—–
    stuff
    more stuff
    etc
    —-END CERTIFICATE—–

    And /var/lib/nagios/clientno/client11-cert.pem should contain both the cert (first) and the CA’s cert (second).

  3. MacLing

    Below is the data:

    $ more LINGGA-cacert.pem
    —–BEGIN CERTIFICATE—–
    snip
    —–END CERTIFICATE—–

    $ more client11-cert.pem
    —–BEGIN CERTIFICATE—–
    snip
    —–END CERTIFICATE—–

    if there is something wrong with the key, why with openssl s_client successfully connected ?
    and i have tested with socat, the key is working properly.

    1. Dan Langille

      I hope you don’t mind, but I amended your post to remove the certificate details. I think not supplying that is a better security decision.

      Yes. I would append LINGGA-cacert.pem to client11-cert.pem. Thus, you would have two certs in client11-cert.pem: first, is the certificate for this server, second is the CA cert.

      Try that.

      If that does not work, my other suggestion: are you using server certificates or client certificates? I always use server certificates. This is the method I use for creating the CSR: <http://www.freebsddiary.org/bacula-tls.php>

  4. sumantra

    Hi Dan,

    I have been trying to setup the “logstash-forwarder” and “logstash” combo. I have been trying to use them in the same machine(win7 64 bit), but would eventually go for a distributed arrangement. Here are the config files that I have been using:

    logstash config:

    input { 
lumberjack {
    # The port to listen on
    host => "localhost"
    port => 5000
 
    # The paths to your ssl cert and key
    ssl_certificate => "C:/Sumantra/logstash-forwarder/logstash-receiver.crt"
    ssl_key         => "C:/Sumantra/logstash-forwarder/logstash-receiver.key"
 
    # Set this to whatever you want.
    type => "MyLogs"
  }
}

output {
  elasticsearch {
    host => localhost
  }
}

forwarder config:

{
  "network": {
    "servers": [ "localhost:5000" ],
 
    "ssl certificate": "C:/Sumantra/logstash-forwarder/logstash-forwarder.crt",
    "ssl key": "C:/Sumantra/logstash-forwarder/logstash-forwarder.key",
 
    "timeout": 15
  },
 
  "files": [
    {
      "paths": [
        "C:/Logs/Core/*"
      ]
    }
  ]
}

    The steps that I am performing are:

    1. starting the forwarder using the command “logstash-forwarder -config forwarder.conf” command.
    2. Starting logstash service and elastic search

    The problem that’s happening is that i keep getting the following message from the forwarder, even after logstash is started:

    2014/09/30 11:50:51.733824 Connecting to localhost:5000
2014/09/30 11:50:52.740925 Failure connecting to localhost:5000: dial tcp 127.0.0.1:5000: 
                           ConnectEx tcp: No connection could be made because the target machine 
                           actively refused it.

    I ran a netstat a couple of times after getting the forwarder started and i received the following information:

    TCP    127.0.0.1:21643        loghost:5000           SYN_SENT        11520
TCP    127.0.0.1:22210        loghost:5000           SYN_SENT        11520

    It seemed to me that the forwarder is TCP-ing to different ports and the logstash process is trying to listen to “5000” and so they are not able to setup some communication. But I may be wrong.

    Could you please help me in finding out what is that I am doing wrong here. Thanks for your help.

Leave a Comment

Scroll to Top