filebeat 部署

发表于 | 分类于

环境信息

使用的 hadoop 完全分布式集群

1
2
3
192.168.2.241 hadoop01 
192.168.2.242 hadoop02 
192.168.2.243 hadoop03

filebeat 安装

官网 https://www.elastic.co/cn/downloads/beats/filebeat

所有节点 root 用户

1
2
3
4
5
6
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.2.0-linux-x86_64.tar.gz

mkdir -p /opt/bigdata/filebeat
tar -zxf filebeat-8.2.0-linux-x86_64.tar.gz -C /opt/bigdata/filebeat
cd /opt/bigdata/filebeat/
ln -s filebeat-8.2.0-linux-x86_64 current

因为已经安装 kafka, 输出到 kafka 中

/opt/bigdata/filebeat/current/filebeat.yml 按需修改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/secure  # 收集登录日志
  fields:
    log_topic: omessages
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
name: "hadoop01"  # 按需修改
setup.kibana:
output.kafka:
  enabled: true
  hosts: ["hadoop01:9092", "hadoop02:9092", "hadoop03:9092"]
  version: "0.10"
  topic: 'my_test'
  codec.format.string: '%{[message]}'  # 输出原始格式， 删除则输出 json 处理后
  partition.round_robin:
    reachable_only: true
  worker: 2
  required_acks: 1
  compression: gzip
  max_message_bytes: 10000000
logging.level: debug
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - drop_fields:  # 删除的字样
      fields: ["input", "host", "agent.type", "agent.ephemeral_id", "agent.id", "agent.version", "ecs"]

测试

root 用户启动 filebeat

1
2
cd /opt/bigdata/filebeat/current
nohup ./filebeat -e -c filebeat.yml &

修改配置后，重启 fiilebeat, ssh 连接主机刷新日志

kafka 显示

1
2
3
4
5
6
7
8
9
10
$ cd /opt/bigdata/kafka/current/bin
$ ./kafka-console-consumer.sh  --bootstrap-server  hadoop01:9092,hadoop02:9092,hadoop03:9092 --topic my_test --from-beginning
{"@timestamp":"2022-05-22T08:20:32.262Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.2.0"},"ecs":{"version":"8.0.0"},"log":{"offset":3204,"file":{"path":"/var/log/secure"}},"message":"May 22 04:20:30 hadoop02 sshd[18047]: pam_systemd(sshd:session): Failed to release session: Interrupted system call","input":{"type":"log"},"host":{"containerized":false,"ip":["192.168.2.242","fe80::ec97:d991:4336:2e98","fe80::f0df:f765:7f99:9634"],"mac":["00:0c:29:68:79:09"],"hostname":"hadoop02","name":"hadoop02","architecture":"x86_64","os":{"type":"linux","platform":"centos","version":"7 (Core)","family":"redhat","name":"CentOS Linux","kernel":"3.10.0-1160.el7.x86_64","codename":"Core"},"id":"0988a88e747e428dbcf4fdc212a6c1ac"},"agent":{"ephemeral_id":"4f44629c-d5e9-4ae4-a5fc-6f96df866dfe","id":"5d7e5f81-16e5-4863-b736-89f6873105ec","name":"hadoop02","type":"filebeat","version":"8.2.0"}}

## 使用 drop_fields
{"@timestamp":"2022-05-22T08:32:49.556Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.2.0"},"log":{"file":{"path":"/var/log/secure"},"offset":5550},"message":"May 22 04:32:49 hadoop02 sshd[23225]: Received disconnect from 192.168.2.242 port 55948:11: disconnected by user","agent":{"name":"hadoop02"}}

## 使用 codec.format.string: '%{[message]}'
May 22 04:43:41 hadoop01 sshd[3106]: Accepted publickey for root from 192.168.2.243 port 41710 ssh2: RSA SHA256:F1RBzp64noGdTdwWX8w+PYfi0zs8ifzkv+etLAOaCJQ
May 22 04:43:41 hadoop01 sshd[3106]: pam_unix(sshd:session): session opened for user root by (uid=0)