Recently I have been working on a project to provision an Elasticsearch cluster for a client. With the recent release of v5.3.1 of the ELK stack, we decided to go ahead and build to that release. Things had been proceeding fairly smoothly, up until this evening when I got hung up with Logstash. The Elasticsearch repo RPM for Logstash is very good about intelligently deciding whether your system is using Upstart, SysVInit, or systemd as an init manager and placing the appropriate startup file.
For Amazon Linux, this happens to be Upstart, and the RPM places a logstash.conf in /etc/init. No problems here, as I’m well aware of how to deal with Upstart vs SysVInit (though I do find it a tad bit strange that 5.3 of the Elasticsearch RPM placed a regular startup script in /etc/init.d, but I suppose that’s a question for another time).
As a simple test, I installed Apache locally on the Logstash host and placed a conf file in /etc/logstash/conf.d/httpd.conf. Interestingly, I never saw an index created or logs pushed to my ES master node. After some experimentation, I noted that if I manually invoke the logstash process with a flag to my conf file, e.g.:
# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/httpd.conf
that logstash would immediately create the index and push docs into the index.
After a bit more experimentation, I came to the conclusion that the problem lay somewhere in how Upstart was invoking the process. I noticed a chroot command in the invocation, which I removed, and bam! The problem goes away, everything is loaded from /etc/logstash, logs are pushed, and I am now a happy camper.
Though I imagine the solution is fairly obvious (remove the chroot command and its parameters), I’ve created a gist with a file that works here (which helps me as well because I am going to leverage it in a Chef cookbook for my implementation in my current project).
Happy logstashing! -b