Rails splunk

Configuring Rails and Splunk (Part 2)

As my wrestle with configuring Splunk continues, I discovered the following helpful article explaining what happens where when forwarding data to a Splunk server.

Basically, it depends. Which is not surprising, given how complicated Splunk is, but it’s good to have on hand!

Combined with details on configuring linebreaking for multi-line events, perhaps I can get my rails logs handled as one entity in the near future…

log4r Rails splunk

Configuring Rails and Splunk (Part 1)

To increase our application intelligence, I’ve started installing Splunk. However, what was not clear to me was the best way of getting my Web/App Rails logs into Splunk. The Web server is on a different box (as is the DB server) so I wanted to know the best way of getting the log data to the Splunk host.

Posts like this imply that Syslog-ng is a good way of getting data to Splunk, but the hidden detail is that if the Splunk server is down, the data is lost. A better way (thanks to the helpful folk on IRC) is to set up a Splunk forwarder on the Web/App/DB hosts and go from there. I went initially with setting them up as (free licenced) Standard Forwarders, so that the web interface is still running for configuration. Remember that any firewall (iptables) will need to be opened for the management ports (8000, 8089 ) and the forward port (9997).

Something else I discovered (but can no longer find in the docs) is that if you configure data sources using the web interface on a Forwarder, but subsequently change the instance to a LightForwarder, you also have to move the configuration files from $SPLUNK_HOME/etc/apps/search/local/ to $SPLUNK_HOME/etc/system/local/