压测实操--kafka-consumer压测方案
作者：九月
环境信息：
操作系统centos7.9，kafka版本为hdp集群中的2.0版本。
Consumer相关参数
使用Kafka自带的kafka-consumer-perf-test.sh脚本进行压测，该脚本参数为：
thread：测试时的单机线程数；
fetch-size：抓取数据量；
partition:分区数，主要是和线程复合来测试；
replication:副本数；
througout:我们所需要的吞吐量，单位时间内处理消息的数量，可能对我们处理消息的延迟有影响；
consumer测试
fetch-size
fetch-size=524288
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size 524288 --threads 4
fetch-size=1048576
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size 1048576 --threads 4
fetch-size=2097152
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size 2097152 --threads 4
fetch-size=3145728
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size 3145728 --threads 4
fetch-size=4194304
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size 4194304 --threads 4
fetch-size=5242880
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 1000000 --fetch-size 5242880 --threads 4
partitions
创建不同partitions的topic
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create --zookeeper mutong01:2181,mutong02:2181,mutong03:2181 --topic test_kafka_consumer1 --partitions 1 --replication-factor 1/usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create --zookeeper mutong01:2181,mutong02:2181,mutong03:2181 --topic test_kafka_consumer2 --partitions 2 --replication-factor 1/usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create --zookeeper mutong01:2181,mutong02:2181,mutong03:2181 --topic test_kafka_consumer3 --partitions 3 --replication-factor 1/usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create --zookeeper mutong01:2181,mutong02:2181,mutong03:2181 --topic test_kafka_consumer4 --partitions 4 --replication-factor 1
topic生产数据
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh --topic test_kafka_consumer1 --num-records 100000000 --record-size 687 --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 batch.size=5000 --throughput 30000/usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh --topic test_kafka_consumer2 --num-records 100000000 --record-size 687 --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 batch.size=5000 --throughput 30000/usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh --topic test_kafka_consumer3 --num-records 100000000 --record-size 687 --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 batch.size=5000 --throughput 30000/usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh --topic test_kafka_consumer4 --num-records 100000000 --record-size 687 --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 batch.size=5000 --throughput 30000
消费不同partitions的topic
消费不同partitions的topic，获取不同partition下的吞吐量
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer1 --messages 10000000 --fetch-size 1048576 --threads 4 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer2 --messages 10000000 --fetch-size 1048576 --threads 4 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer3 --messages 10000000 --fetch-size 1048576 --threads 4 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer4 --messages 10000000 --fetch-size 1048576 --threads 4
fetch-threads
fetch-thread=1
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576 --messages 10000000 --threads 1 --num-fetch-threads 1
fetch-thread=4
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576 --messages 10000000 --threads 1 --num-fetch-threads 4
fetch-thread=7
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576 --messages 10000000 --threads 1 --num-fetch-threads 7
fetch-thread=10
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576 --messages 10000000 --threads 1 --num-fetch-threads 10
总结
使用kafka自带的kafka-consumer-perf-test.sh脚本对集群消费情况进行压测，根据业务情况设置要测试的messager条数。不同的变量因素，通过修改其中一个变量条件，固定其余变量值，获取对应的吞吐速率。整理不同条件下的指标结果，形成曲线就可以观察具体的消费趋势。
