转载于http://www.eryajf.net/2369.html
上边是输出了nginx日志,从而进行展示,以及各种绘图分析,而现在的需求是,要将微服务当中的日志汇总到elk当中以便开发查询日志定位问题。
都知道,微服务第一个特点就是,多,不仅项目多,而且往往单台主机当中也会有多个应用,因此多个日志文件情况下,如何处理才更加快速便捷呢,这里使用了filebeat来作为日志转发组件。
架构如图:
主机规划如下图简示:
主机 | 组件 |
---|---|
192.168.100.21 | spring-cloud,filebeat-6.5.3 |
192.168.100.21 | spring-cloud,filebeat-6.5.3 |
192.168.10.10 | logstash-6.5.3,elk |
像刚刚那样,配置好yun源,然后直接安装。
yum -y install filebeat
然后来配置filebeat。
cat > /etc/filebeat/filebeat.yml << EOF
filebeat.inputs:
- input_type: log
paths:
- /home/ishangjie/ishangjie-config-server/normal/*.log
type: "wf1-config"
fields:
logsource: 192.168.100.21
logtype: wf1-config
- input_type: log
paths:
- /home/ishangjie/ishangjie-eureka-server/normal/*.log
type: "wf1-eureka"
fields:
logsource: 192.168.100.21
logtype: wf1-eureka
- input_type: log
paths:
- /home/ishangjie/ishangjie-gateway-server/normal/*.log
type: "wf1-gateway"
fields:
logsource: 192.168.100.21
logtype: wf1-gateway
output.logstash:
hosts: ["192.168.10.10:5044"]
EOF
再去配置一下另外一台主机。
cat > /etc/filebeat/filebeat.yml << EOF
filebeat.inputs:
- input_type: log
paths:
- /home/ishangjie/ishangjie-activity-service/normal/*.log
type: "wf5-activity"
fields:
logsource: 192.168.100.25
logtype: wf5-activity
- input_type: log
paths:
- /home/ishangjie/ishangjie-order-service/normal/*.log
type: "wf5-order"
fields:
logsource: 192.168.100.25
logtype: wf5-order
- input_type: log
paths:
- /home/ishangjie/ishangjie-user-service/normal/*.log
type: "wf5-user"
fields:
logsource: 192.168.100.25
logtype: wf5-user
- input_type: log
paths:
- /home/ishangjie/ishangjie-thirdparty-service/normal/*.log
type: "wf5-thirdparty"
fields:
logsource: 192.168.100.25
logtype: wf5-thirdparty
output.logstash:
hosts: ["192.168.10.10:5045"]
EOF
启动filebeat。
systemctl enable filebeat
systemctl start filebeat
systemctl status filebeat
针对上边两个主机转过来的日志,在elk主机上添加相对应的配置进行接收。
A
:
cat > /etc/logstash/conf.d/wf1.conf << EOF
input {
beats {
port => "5044"
host => "192.168.100.21"
}
}
filter {
if [fields][logtype] == "wf1-config" {
json {
source => "message"
target => "data"
}
}
if [fields][logtype] == "wf1-eureka" {
json {
source => "message"
target => "data"
}
}
if [fields][logtype] == "wf1-gateway" {
json {
source => "message"
target => "data"
}
}
}
output {
if [fields][logtype] == "wf1-config" {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "wf1-config-%{+YYYY.MM.dd}"
}
}
if [fields][logtype] == "wf1-eureka" {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "wf1-eureka-%{+YYYY.MM.dd}"
}
}
if [fields][logtype] == "wf1-gateway" {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "wf1-gateway-%{+YYYY.MM.dd}"
}
}
}
EOF
B
:
cat > /etc/logstash/conf.d/wf5.conf << EOF
input {
beats {
port => 5052
host => "192.168.100.25"
}
}
filter {
if [fields][logtype] == "wf5-activity" {
json {
source => "message"
target => "data"
}
}
if [fields][logtype] == "wf5-order" {
json {
source => "message"
target => "data"
}
}
if [fields][logtype] == "wf5-user" {
json {
source => "message"
target => "data"
}
}
if [fields][logtype] == "wf5-thirdparty" {
json {
source => "message"
target => "data"
}
}
}
output {
if [fields][logtype] == "wf5-activity" {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "wf5-activity-%{+YYYY.MM.dd}"
}
}
if [fields][logtype] == "wf5-order" {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "wf5-order-%{+YYYY.MM.dd}"
}
}
if [fields][logtype] == "wf5-user" {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "wf5-user-%{+YYYY.MM.dd}"
}
}
if [fields][logtype] == "wf5-thirdparty" {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "wf5-thirdparty-%{+YYYY.MM.dd}"
}
}
}
EOF
启动这两个实例。
nohup /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/wf1.conf --path.data=/usr/share/logstash/data5 &> /logs/logstash_nohup/wf1.out &
nohup /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/wf5.conf --path.data=/usr/share/logstash/data9 &> /logs/logstash_nohup/wf5.out &
启动之后可以按上边演示过的步骤,在kibana当中添加索引,然后查看日志。
ELK-6.5.3学习笔记–使用filebeat管理微服务日志
原文:https://www.cnblogs.com/cheyunhua/p/10755952.html