bigdata监控

发表于 | 分类于

环境信息

使用的 hadoop 完全分布式集群

1
2
3
192.168.2.241 hadoop01 
192.168.2.242 hadoop02 
192.168.2.243 hadoop03

kafka 监控

主要有 Kafka Manager、Kafka Eagle。 当前使用 Kafka Eagle

官网 https://www.kafka-eagle.org/

hadoopclient 安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
wget https://github.com/smartloli/kafka-eagle-bin/archive/v2.1.0.tar.gz

mkdir -p /opt/bigdata/kafka-eagle
tar -zxf v2.1.0.tar.gz
cd kafka-eagle-bin-2.1.0
tar zxf efak-web-2.1.0-bin.tar.gz -C /opt/bigdata/kafka-eagle
cd /opt/bigdata/kafka-eagle
ln -s  efak-web-2.1.0 current
chown -R kafka:kafka /opt/bigdata/kafka-eagle

cat >/etc/profile.d/kafka_env.sh<<eof
export KE_HOME=/opt/bigdata/kafka-eagle/current/
export PATH=$PATH:$KE_HOME/bin
export JAVA_HOME=/opt/bigdata/java/current
eof
source /etc/profile

配置数据库

1
2
3
4
5
6
7
8
9
10
11
12
13
mysql>  create database ke character set utf8mb4;
Query OK, 1 row affected (0.00 sec)

mysql>  create user 'root'@'192.168.2.86' identified by 'kafkapass';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on ke.* to  'root'@'192.168.2.86';
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql>

修改配置文件
/opt/bigdata/kafka-eagle/current/conf/system-config.properties

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
efak.zk.cluster.alias=cluster1,cluster2
cluster1.zk.list=192.168.2.241:2181,192.168.2.242:2181,192.168.2.243:2181
cluster2.zk.list=192.168.2.171:2181,192.168.2.173:2181,192.168.2.174:2181
kafka.zk.limit.size=25
efak.webui.port=8048
cluster1.kafka.efak.offset.storage=kafka
cluster2.kafka.efak.offset.storage=kafka
efak.metrics.charts=true
efak.metrics.retain=15
efak.sql.topic.records.max=5000
efak.sql.fix.error=true
efak.topic.token=keadmin
efak.driver=com.mysql.jdbc.Driver
efak.url=jdbc:mysql://192.168.2.86:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
efak.username=root
efak.password=kafkapass

修改 kafka 的启动脚本 /opt/bigdata/kafka/current/bin/kafka-server-start.sh

1
2
3
4
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    export JMX_PORT="9999"  # 添加此项，因为开启了监控趋势图 efak.metrics.charts=true
fi

重启 kafka

1
2
3
cd /opt/bigdata/kafka/current/bin/
bash kafka-server-stop.sh # 确认关闭后,执行启动
nohup bin/kafka-server-start.sh  config/server.properties &

启动 Kafka Eagle

1
2
cd /opt/bigdata/kafka-eagle/current
./bin/ke.sh  start

验证

EFAK测试