Setting Up Logstash for ELK

scottalanmiller

If you use nearly any logstash-forwarder documentation that you find, you might find that the information around CentOS and RHEL is a bit out of date. This can make things very challenging. The most popular documentation around is from Digital Ocean and their Ubuntu docs have been updated but their CentOS have not. With a little work and research I was able to come up with a script that does everything that you need for CentOS and RHEL.

#!/bin/bash
#Set Up ELK

mkdir -p /etc/pki/tls/certs
echo '-----BEGIN CERTIFICATE-----
....contents of your cert go here.....
-----END CERTIFICATE-----' > /etc/pki/tls/certs/logstash-forwarder.crt

wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm
yum -y install logstash-forwarder-0.4.0-1.x86_64.rpm
rm logstash-forwarder-0.4.0-1.x86_64.rpm

echo '{
  "network": {
    "servers": [ "1.2.3.4:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [
        "/var/log/messages",
        "/var/log/security"
       ],
      "fields": { "type": "syslog" }
    }
   ]
}' > /etc/logstash-forwarder.conf

chkconfig --add logstash-forwarder
service logstash-forwarder start

You will need to paste in the contents of your own key, of course and the 1.2.3.4 needs to be changed to your ELK's IP address. But other than that, you can just use this script and you are ready to go. Tested on CentOS 6 and CentOS 7. This will get basic logs flowing into a Digital Ocean style ELK install that is currently up to date (Kibana 4 era.)

scottalanmiller

If you have older CentOS 5 or RHEL 5 you can still use the script, this has been tested and all that we need to is remove the final line of the script and replace it with...

/etc/init.d/logstash-forwarder start

And now you have CentOS 5, 6 and 7 all supported.

JaredBusch

I believe these two commands are not available by default in CentOS 7.

chkconfig --add logstash-forwarder
service logstash-forwarder start

should be

systemctl enable logstash-forwarder
systemctl start logstash-forwarder

scottalanmiller

@JaredBusch said:

I believe these two commands are not available by default in CentOS 7.
chkconfig --add logstash-forwarder
service logstash-forwarder start
should be
systemctl enable logstash-forwarder
systemctl start logstash-forwarder

Those are the proper ones, definitely. The old ones still work in 7, probably gone in 8, though. The script worked repeatedly on several 7 machines. We only have two 6s left in the fleet, I think.

stacksofplates

So I just finished installing again on CentOS 7. I'm having an issue which is the same one I had before.

I have a suspicion that it has to do with filebeat. I used Digital Ocean's doc back in October or so and everything worked fine. Now that they switched to filebeat it's not working.

scottalanmiller

Great. Going to have to do some research. I am planning on building a new one soon myself.

stacksofplates

So it's not SELinux or firewalld, both are completely off on both the client and the ELK server.

stacksofplates

Eh I'm just a giant moron. Port 5044 wasn't open, and stopping firewalld doesn't stop the firewall (idiot) so now it's working.

They don't do that in the documentation though, so just remember to do it. I remembered about port 80 for kibana, but forgot about logstash.

stacksofplates

They also forget about SELinux with their CentOS 7 docs. You need sudo setsebool -P httpd_can_network_connect 1 and possibly sudo chcon -R --type=httpd_syscontent_rw_t /opt/kibana

Up and running now.