Logstash shipper configuration for Apache access log and error log

Advertisements

Logstash is one of the best tool for the log analysis. We can create pie charts bar graphs and a lot of advanced charts using Logstash and Kibana.

We all use Apache webservers. How to send apache web server logs to Logstash? Easiest way is to use the logstash shipper.

You can use the following configuration file to ship apache access and error logs to a redis server which acts as a aggregator for logstash.

[root@test patterns]# cat /etc/logstash/shipper.conf
input {
file {
        start_position => beginning
        path => ["/var/log/httpd/access_log"]
        type => "apache-access"
        }
file {
        start_position => beginning
        path => ["/var/log/httpd/error_log"]
        type => "apache-error"
        }
}

filter {
 if [type] == "apache-access" {
  grok {
        match => { "message" => "%{COMBINEDAPACHELOG}" }
        add_tag => [ "vod_origin" ]
        }
}
 if [type] == "apache-error" {

  grok {
        patterns_dir => [ "/etc/logstash/patterns" ]
        match => [ "message", "%{APACHE_ERROR_LOG}" ]
        add_tag => [ "vod_origin" ]
        }
}
 geoip {
        source => "clientip"
        }
}

output {
stdout { }
redis {
        host => "xxx.xxx.xxx.xxx"
        data_type => "list"
        key => "logstash"
        }
}

[root@testpatterns]# cat /etc/logstash/patterns/apache-error
APACHE_ERROR_TIME %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
APACHE_ERROR_LOG \[%{APACHE_ERROR_TIME:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg}