当前位置：首页 > news >正文

JMXtrans + InfluxDB + Grafana实现Kafka性能指标监控

news 来源：原创 2024/5/3 19:15:16

2019独角兽企业重金招聘Python工程师标准>>>

架构

一般系统监控通常分为3部分：

数据采集
分析与转换
展现(可视化)

数据采集

对于前端应用，一般需要埋点，对用户的行为进行记录。如果不埋点，则需要通过Pagespeed、PhantomJS这样的工具去模拟用户行为进行测试。后端的系统通常有自己的性能指标。我们可以通过命令／脚本的方式进行采集。

JMX（Java Management Extensions，即Java管理扩展）是一个为应用程序、设备、系统等管理功能的框架，通常可以用来监控和管理Java应用系统。

Kafka做为一款Java应用，已经定义了丰富的性能指标，(可以参考 Kafka监控指标 )，通过JMX可以轻松对其进行监控。

测试

首先需要在Kafka上打开JMX

1.修改${kafka_home}/bin/kafka-server-start.sh , 增加一个JMX_PORT的配置，指定一个端口用于接受外部连接，注意如部署、运行在非root用户下，必须指定1024以上端口

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    export JMX_PORT="9999"
fi

2.重启kafka

bin/kafka-server-stop.sh
bin/kafka-server-start.sh config/server.properties &

3.重启后观察可以发现JMX已经启动了

$ps -ef | grep kafka
www       51065      1  0 Nov03 ?        00:08:20 /opt/vdian/java/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/home/www/kafka_2.11-0.10.0.1/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9999 -Dkafka.logs.dir=/home/www/kafka_2.11-0.10.0.1/bin/../logs -Dlog4j.configuration=file:bin/../config/log4j.properties -cp :/home/www/kafka_2.11-0.10.0.1/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/argparse4j-0.5.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/connect-api-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/connect-file-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/connect-json-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/connect-runtime-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/guava-18.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/hk2-api-2.4.0-b34.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/hk2-locator-2.4.0-b34.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/hk2-utils-2.4.0-b34.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-annotations-2.6.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-core-2.6.3.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-databind-2.6.3.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javassist-3.18.2-GA.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javax.annotation-api-1.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javax.inject-1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javax.inject-2.4.0-b34.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javax.servlet-api-3.1.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-client-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-common-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-container-servlet-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-guava-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-media-jaxb-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-server-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jopt-simple-4.9.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka-clients-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka-log4j-appender-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka-streams-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka-streams-examples-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka-tools-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1-sources.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1-test-sources.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/log4j-1.2.17.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/lz4-1.3.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/metrics-core-2.2.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/reflections-0.9.10.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/rocksdbjni-4.8.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/scala-library-2.11.8.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/slf4j-api-1.7.21.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/slf4j-log4j12-1.7.21.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/snappy-java-1.1.2.6.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/validation-api-1.1.0.Final.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/zkclient-0.8.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/zookeeper-3.4.6.jar kafka.Kafka config/server.properties
 
$ netstat -an | grep 9999
tcp        0      0 :::9999                     :::*                        LISTEN

4.在安装了java的主机上运行jconsole，就会弹出一个控制台，在可以看到MBean中的性能指标

636NRfb.png!web Zn26Zzi.png!web

指标采集

传统的数据采集方案，一般是先开发数据采集的脚本，然后借助nagios、zabbix这样的监控软件来调度执行，并将采集到的数据进行上报。对于java应用，给大家介绍一个新朋友jmxtrans。

读取json或yaml格式的配置文件，通过jmx采集java性能指标。支持输出到Graphite、InfluxDB、RRDTool等。

安装部署

1.首先下载jmxtrans的RPM包，地址

2.安装jdk1.8

yum install -y java-1.8.0-openjdk.x86_64 java-1.8.0-openjdk-devel.x86_64

3.设置JAVA_HOME和PATH环境变量

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-0.b17.el6_7.x86_64
export PATH=$JAVA_HOME/bin:/usr/share/jmxtrans/bin:$PATH

4.编写配置文件

下文是一段测试用的配置，采集的数据会输出到日志中显示。

{
  "servers" : [ {
    "port": "jmx端口",
    "host": "IP地址",
    "queries" : [ {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.StdOutWriter"
      } ],
      "obj" : "java.lang:type=Memory",
      "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ]
    }, {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.StdOutWriter"
      } ],
      "obj" : "java.lang:name=CMS Old Gen,type=MemoryPool",
      "attr" : [ "Usage" ]
    }, {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.StdOutWriter"
      } ],
      "obj" : "java.lang:name=ConcurrentMarkSweep,type=GarbageCollector",
      "attr" : [ "LastGcInfo" ]
    } ],
    "numQueryThreads" : 2
  } ]
}

5.配置文件编辑完成后，将其放在 /var/lib/jmxtrans/ 目录中

6.启动jmxtrans