logstash分析httpd_log

httpd或者nginx格式

logstash内置支持的格式有两种，兼容httpd的common 和 combined.

COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

相当于apache httpd的:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

LogFormat "%h %l %u %t \"%r\" %>s %b" common

相当于nginx上面的main 去掉”$http_x_forwarded_for” :

  log_format combined ‘$remote_addr - $remote_user [$time_local] ‘
                    ‘"$request" $status $body_bytes_sent ‘
                    ‘"$http_referer" "$http_user_agent"‘;

被采集机器上配置logstash,输出到elasticsearch机器的redis上去。

input {
        file {
                type => "apache_log"
                path => ["/var/log/httpd/access_log"]
        }
}
output {
        redis {
                host => "xx.xx.xx.xx"
                data_type => "list"
                key => "logstash:redis"

        }
    stdout { codec => rubydebug }
}

这里确保机器能正确连接到redis的端口。
telnet IP 6397

elasticsearch机器将redis队列里面的内容读取到elasticsearch上去:

input
{
 redis {
    host => "127.0.0.1"
    data_type => "list"
    key => "logstash:redis"
  }
}
filter
{
    grok {
        match => { "message" => "%{COMMONAPACHELOG:apachelog}" }
        add_field => [ "response", "%{NUMBER:response}" ]
    }
}
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }

}

之后就是kibana显示了。

略