为什么80%的码农都做不了架构师？>>> 
磁盘I/O 子系统是Linux 系统中最慢的部分,当觉得系统中出现了I/O 瓶颈时，如何监控系统的I/O情况，本文介绍的iostat命令完全可以胜任这个工作，iostat命令是报告cpu的统计信息和磁盘的i/o统计信息，下面从man手册来学习这个命令
NAME
iostat – Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.
iostat命令是报告cpu的统计信息和磁盘的i/o统计信息
iostat [ -c ] [ -d ] [ -h ] [ -N ] [ -k | -m ] [ -t ] [ -V ] [ -x ] [ -z ] [ device [...] | ALL ] [ -p [ device [,...] | ALL ] ] [ interval [ count ] ]DESCRIPTION
The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates. The iostat command generates reports that can be used to change system configuration to better balance the input/output load between physical disks.
iostat命令通过观察存储设备实际的工作时间和它们的平均传输率来监控系统的i/o负载。iostat命令生成的报告可以用来改变系统配置来更好的平衡各个磁盘的i/o负载
The first report generated by the iostat command provides statistics concerning the time since the system was booted. Each subsequent report covers the time since the previous report. All statistics are reported each time the iostat command is run. The report consists of a CPU header row followed by a row of CPU statistics. On multiprocessor systems, CPU statistics are calculated system-wide as averages among all processors. A device header row is displayed followed by a line of statistics for each device that is configured.
iostat命令生成的第一个报告的统计信息的时间是从系统启动时开始算起的。每一个后面的报告都包含了之前报告的时间。所有的统计数据都是每一个 iostat命令运行统计出来的。报告包括一行cpu信息标题字段和一行cpu的统计信息。在多cpu系统中，cpu的统计信息是所有cpu的平均值。 io系统标题行后面每一行统计数据都是每个i/o设备的统计信息
The interval parameter specifies the amount of time in seconds between each report. The first report contains statistics for the time since system startup (boot). Each subsequent report contains statisics collected during the interval since the previous report. The count parameter can be specified in conjunction with the interval parameter. If the count parameter is specified, the value of count determines the number of reports generated at interval seconds apart. If the interval parameter is specified without the count parameter, the iostat command generates reports continuously.
间隔参数是指每个报告之间的时间间隔,以秒为单位。第一个报告包含系统启动以来的统计数据。每个后续报告包含自上次报告的时间和间隔期间收集的统计 信息的累积量。计数参数可以间隔参数一起使用。如果计数参数指定了，计数的值决定了生成的报告的数量。如果计数参数没有指定，则iostat命令会一直按 指定的时间间隔生产报告
REPORTS
The iostat command generates two types of reports, the CPU Utilization report and the Device Utilization report.
iostat命令生产两种类型的报告，cpu利用率报告和磁盘利用率报告
CPU Utilization Re
The first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors. The report has the following format:
%user Show the percentage of CPU utilization that occurred while executing at the user level (application).
在用户级别运行所使用的CPU的百分比.%nice Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
优先进程消耗的CPU时间，占所有CPU的百分比%system Show the percentage of CPU utilization that occurred while executing at the system level (kernel).
在系统级别(kernel)运行所使用CPU的百分比.%iowait Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
CPU等待硬件I/O时,所占用CPU百分比%steal Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
管理程序维护另一个虚拟处理器时，虚拟CPU的无意识等待时间百分比。%idle Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
CPU空闲时间的百分比Device Utilization Report
The second report generated by the iostat command is the Device Utilization Report. The device report provides statistics on a per physical device or partition basis. Block devices for which statistics are to be displayed may be entered on the command line. Partitions may also be entered on the command line providing that option -x is not used. If no device nor partition is entered, then statistics are displayed for every device used by the system, and providing that the kernel maintains statistics for it. If the ALL keyword is given on the command line, then statistics are displayed for every device defined by the system, including those that have never been used. The report may show the following fields, depending on the flags used:
此报告是iostat命令生成的i/o设备利用率报告。报告提供了每一个物理存储设备或分区的统计信息。通过命令行可以指定显示某个块设备的统计信息。如 果不用-x选项是也可以用命令指定显示某个分区的统计信息。如果不在命令中指定块设备或分区，将显示被系统识别的每一个设备的统计信息，包括那些那些从来 没有使用的设备。报告根据命令行的选项可以显示其他的字段信息：
Device:
This column gives the device (or partition) name, which is displayed as hdiskn with 2.2
kernels, for the nth device. It is displayed as devm-n with 2.4 kernels, where m is the
major number of the device, and n a distinctive number. With newer kernels, the device
name as listed in the /dev directory is displayed.
tps Indicate the number of transfers per second that were issued to the device. A transfer is an I/O request to the device. Multiple logical requests can be combined into a sin-
gle I/O request to the device. A transfer is of indeterminate size.
显示设备的每秒传输次数。一次传输就是一次i/o请求。多个逻辑请求可以合并成一次i/o请求。
每次传输的数据量大小不能确定
Blk_read/s Indicate the amount of data read from the device expressed in a number of blocks per
second. Blocks are equivalent to sectors with 2.4 kernels and newer and therefore have
a size of 512 bytes. With older kernels, a block is of indeterminate size.
每秒读取的数据块数。在2.4内核及以上版本的内核数据块等于磁盘扇区的大小512bytes，
老的内核版本数据块的大学不确定。
Blk_wrtn/s Indicate the amount of data written to the device expressed in a number of blocks per second.
每秒写入的数据块数
Blk_read
The total number of blocks read.
从系统启动到现在读入的数据块数
Blk_wrtn
The total number of blocks written.
从系统启动到现在写入的数据块数
kB_read/s Indicate the amount of data read from the device expressed in kilobytes per second.
每秒从设备读取多少kb数据量
kB_wrtn/s Indicate the amount of data written to the device expressed in kilobytes per second.
每秒写入设备多少kb数据量
kB_read The total number of kilobytes read.
从系统启动到现在总共读取了多少kb数据量
kB_wrtn The total number of kilobytes written.
从系统启动到现在总共写入了多少kb数据量
MB_read/s Indicate the amount of data read from the device expressed in megabytes per second.
每秒从设备读取多少mb数据量
MB_wrtn/s Indicate the amount of data written to the device expressed in megabytes per second.
每秒写入设备多少mb数据量
MB_read The total number of megabytes read.
从系统启动到现在总共读取了多少mb数据量
MB_wrtn The total number of megabytes written.
从系统启动到现在总共写入了多少mb数据量
rrqm/s The number of read requests merged per second that were queued to the device.
将读入请求合并后,每秒发送到设备的读入请求数
wrqm/s The number of write requests merged per second that were queued to the device.
将写入请求合并后,每秒发送到设备的写入请求数
r/s The number of read requests that were issued to the device per second.
每秒发送到设备的读入请求数
w/s The number of write requests that were issued to the device per second.
每秒发送到设备的写入请求数
rsec/s The number of sectors read from the device per second.
每秒从设备读入的扇区数
wsec/s The number of sectors written to the device per second.
每秒向设备写入的扇区数.
rkB/s The number of kilobytes read from the device per second.
wkB/s The number of kilobytes written to the device per second.
rMB/s The number of megabytes read from the device per second.
wMB/s The number of megabytes written to the device per second.
avgrq-sz The average size (in sectors) of the requests that were issued to the device.
发送到设备的请求的平均大小,单位是扇区
avgqu-sz The average queue length of the requests that were issued to the device.
发送到设备的请求的平均队列长度
await The average time (in milliseconds) for I/O requests issued to the device to be served.
This includes the time spent by the requests in queue and the time spent servicing them.
I/O请求平均执行时间.包括发送请求和执行的时间.单位是毫秒
svctm The average service time (in milliseconds) for I/O requests that were issued to the device.
发送到设备的I/O请求的平均执行时间.单位是毫秒
%util Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device).
Device saturation occurs when this value is close to 100%.
在I/O请求发送到设备期间,占用CPU时间的百分比.用于显示设备的带宽利用率，当这个值接近100%时,表示设备带宽已经占满
rops/s Indicate the number of read operations that were issued to the mount point per second 挂载点每秒多少次读操作
wops/s Indicate the number of write operations that were issued to the mount point per second 挂载点多少次写操作OPTIONS 参数
-c The -c option is exclusive of the -d option and displays only the CPU usage report.
仅显示CPU统计信息.与-d选项互斥.-d The -d option is exclusive of the -c option and displays only the device utilization report.
仅显示磁盘统计信息.与-c选项互斥-k Display statistics in kilobytes per second instead of blocks per second.
Data displayed are valid only with kernels 2.4 and newer.
以K为单位显示每秒的磁盘请求数,默认单位块-m Display statistics in megabytes per second instead of blocks or kilobytes per second.
Data displayed are valid only with kernels 2.4 and newer.
以M为单位显示每秒的磁盘请求数,默认单位块-n Displays the NFS-directory statistic. Data displayed are valid only with kernels 2.6.17 and
newer. This option is exclusive ot the -x option.
显示nfs统计信息-h Display the NFS report more human readable.
-p [ { device | ALL } ]
The -p option is exclusive of the -x option and displays statistics for block devices and all
their partitions that are used by the system. If a device name is entered on the command
line, then statistics for it and all its partitions are displayed. Last, the ALL keyword indi-
cates that statistics have to be displayed for all the block devices and partitions defined by
the system, including those that have never been used. Note that this option works only with
post 2.5 kernels.
与-x选项互斥,用于显示块设备及系统分区的统计信息-t Print the time for each report displayed.
在输出数据时,打印搜集数据的时间-V Print version number then exit.-x Display extended statistics. This option is exclusive of the -p and -n, and works with post 2.5 kernels since it needs /proc/diskstats file or a mounted sysfs to get the statistics. This
option may also work with older kernels (e.g. 2.4) only if extended statistics are available in /proc/partitions (the kernel needs to be patched for that).
输出扩展信息示例
参数 -d 表示，显示设备（磁盘）使用状态；-k某些使用block为单位的列强制使用Kilobytes为单位；2表示，数据显示每隔2秒刷新一次。
# iostat // 显示一条统计记录,包括所有的CPU和设备.
[root@rac1 oswbb]# iostatLinux 2.6.18-164.el5 (rac1) 01/23/2015avg-cpu: %user %nice %system %iowait %steal %idle 3.43 0.05 10.80 10.66 0.00 75.07Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtnsda 4.33 14.94 129.96 2693442 23426434sda1 0.00 0.01 0.00 2402 10sda2 3.36 13.10 119.21 2361759 21489488sda3 0.98 1.82 10.75 328913 1936936# iostat -d 2 // 每隔2秒,显示一次设备统计信息.
[root@rac1 oswbb]# iostat -d 2Linux 2.6.18-164.el5 (rac1) 01/23/2015Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtnsda 4.33 14.88 129.83 2693466 23502386sda1 0.00 0.01 0.00 2402 10sda2 3.35 13.05 119.09 2361783 21557728sda3 0.98 1.82 10.74 328913 1944648# iostat -d 2 6 // 每隔2秒,显示一次设备统计信息.总共输出6次.
[root@rac1 oswbb]# iostat -d 2Linux 2.6.18-164.el5 (rac1) 01/23/2015Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtnsda 4.33 14.88 129.83 2693466 23502386sda1 0.00 0.01 0.00 2402 10sda2 3.35 13.05 119.09 2361783 21557728sda3 0.98 1.82 10.74 328913 1944648Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtnsda 4.33 14.88 129.83 2693466 23502386sda1 0.00 0.01 0.00 2402 10sda2 3.35 13.05 119.09 2361783 21557728sda3 0.98 1.82 10.74 328913 1944648# iostat -x sda sdb 2 6// 每隔2秒显示一次hda,hdb两个设备的扩展统计信息,共输出6次.
[root@rac1 oswbb]# iostat -x sda sdb 2 6Linux 2.6.18-164.el5 (rac1) 01/23/2015avg-cpu: %user %nice %system %iowait %steal %idle 3.42 0.05 10.79 10.66 0.00 75.07Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.19 12.36 0.46 3.86 14.87 129.80 33.44 0.02 4.85 1.38 0.60sdb 0.00 0.00 0.11 0.26 1.49 2.11 9.51 0.00 8.95 8.58 0.32# iostat -p sda 2 6 // 每隔2秒显示一次sda及上面所有分区的统计信息,共输出6次.
[root@rac1 oswbb]# iostat -p sda 2 6 Linux 2.6.18-164.el5 (rac1) 01/23/2015avg-cpu: %user %nice %system %iowait %steal %idle 3.42 0.05 10.79 10.66 0.00 75.07Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtnsda 4.33 14.87 129.79 2693474 23515370sda3 1.86 0.01 0.98 2092 178453sda2 13.45 0.17 9.88 30717 1789685sda1 0.01 0.01 0.02 1070 3759下面是oswbb通过iostat -xk输出iostat信息片段
zzz ***Mon Jan 19 15:00:10 CST 2015avg-cpu: %user %nice %system %iowait %steal %idle 0.15 0.00 0.15 0.00 0.00 99.70Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.00 46.00 0.00 2.00 0.00 192.00 192.00 0.00 0.00 0.00 0.00cciss/c0d0p1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00cciss/c0d0p2 0.00 46.00 0.00 2.00 0.00 192.00 192.00 0.00 0.00 0.00 0.00cciss/c0d0p3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00cciss/c0d0p4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00cciss/c0d0p5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00cciss/c0d0p6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00cciss/c0d0p7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00%iowait并不能反应磁盘瓶颈
iowait实际测量的是cpu时间：%iowait = (cpu idle time)/(all cpu time)
这个文章说明：高速cpu会造成很高的iowait值，但这并不代表磁盘是系统的瓶颈。唯一能说明磁盘是系统瓶颈的方法，就是很高的read/write时间，一般来说超过20ms，就代表了不太正常的磁盘性能。为什么是20ms呢？一般来说，一次读写就是一次寻到+一次旋转延迟+数据传输的时间。由于，现代硬盘数据传输就是几微秒或者几十微秒的事情，远远小于寻道时间2~20ms和旋转延迟4~8ms，所以只计算这两个时间就差不多了，也就是15~20ms。只要大于20ms，就必须考虑是否交给磁盘读写的次数太多，导致磁盘性能降低了。
作者的文章以AIX系统为例，使用其工具filemon来检测磁盘每次读写平均耗时。在Linux下，可以通过iostat命令还查看磁盘性能。其 中的svctm一项，反应了磁盘的负载情况，如果该项大于15ms，并且util%接近100%，那就说明，磁盘现在是整个系统性能的瓶颈了。
iostat来对linux硬盘IO性能进行了解
以前一直不太会用这个参数。现在认真研究了一下iostat，因为刚好有台重要的服务器压力高,所以放上来分析一下.下面这台就是IO有压力过大的服务器
[root@rac1 oswbb]# iostat -x 1 1Linux 2.6.18-164.el5 (rac1) 01/23/2015avg-cpu: %user %nice %system %iowait %steal %idle 3.42 0.05 10.79 10.65 0.00 75.08Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.19 12.34 0.46 3.86 14.81 129.64 33.44 0.02 4.84 1.37 0.59字段描述
rrqm/s:每秒进行 merge 的读操作数目。即 delta(rmerge)/s
wrqm/s:每秒进行 merge 的写操作数目。即 delta(wmerge)/s
r/s:每秒完成的读 I/O 设备次数。即 delta(rio)/s
w/s:每秒完成的写 I/O 设备次数。即 delta(wio)/s
rsec/s:每秒读扇区数。即 delta(rsect)/s
wsec/s:每秒写扇区数。即 delta(wsect)/s
rkB/s:每秒读K字节数。是 rsect/s 的一半，因为每扇区大小为512字节。(需要计算)wkB/s:每秒写K字节数。是 wsect/s 的一半。(需要计算)avgrq-sz:平均每次设备I/O操作的数据大小 (扇区)。delta(rsect+wsect)/delta(rio+wio)avgqu-sz:平均I/O队列长度。即 delta(aveq)/s/1000 (因为aveq的单位为毫秒)。await:平均每次设备I/O操作的等待时间 (毫秒)。即 delta(ruse+wuse)/delta(rio+wio)svctm:平均每次设备I/O操作的服务时间 (毫秒)。即 delta(use)/delta(rio+wio)%util: 一秒中有百分之多少的时间用于 I/O 操作，或者说一秒中有多少时间 I/O 队列是非空的。
即 delta(use)/s/1000 (因为use的单位为毫秒)如果 %util 接近 100%，说明产生的I/O请求太多，I/O系统已经满负荷，该磁盘可能存在瓶颈。
idle小于70% IO压力就较大了,一般读取速度有较多的wait。
同时可以结合vmstat 查看查看b参数(等待资源的进程数)和wa参数(IO等待所占用的CPU时间的百分比,高过30%时IO压力高)
另外 await 的参数也要多和 svctm 来参考。差的过高就一定有 IO 的问题。
avgqu-sz 也是个做 IO 调优时需要注意的地方，这个就是直接每次操作的数据的大小，如果次数多，但数据拿的小的话，其实 IO 也会很小.如果数据拿的大，才IO 的数据会高。也可以通过 avgqu-sz × ( r/s or w/s ) = rsec/s or wsec/s.也就是讲，读定速度是这个来决定的。
另外还可以参考svctm 一般要小于 await (因为同时等待的请求的等待时间被重复计算了)，svctm 的大小一般和磁盘性能有关，CPU/内存的负荷也会对其有影响，请求过多也会间接导致 svctm 的增加。await 的大小一般取决于服务时间(svctm) 以及 I/O 队列的长度和 I/O 请求的发出模式。如果 svctm 比较接近 await，说明 I/O 几乎没有等待时间；如果 await 远大于 svctm，说明 I/O 队列太长，应用得到的响应时间变慢，如果响应时间超过了用户可以容许的范围，这时可以考虑更换更快的磁盘，调整内核 elevator 算法，优化应用，或者升级 CPU。
队列长度(avgqu-sz)也可作为衡量系统 I/O 负荷的指标，但由于 avgqu-sz 是按照单位时间的平均值，所以不能反映瞬间的 I/O 洪水。
别人一个不错的例子(I/O 系统 vs. 超市排队)
举一个例子，我们在超市排队 checkout 时，怎么决定该去哪个交款台呢? 首当是看排的队人数，5个人总比20人要快吧? 除了数人头，我们也常常看看前面人购买的东西多少，如果前面有个采购了一星期食品的大妈，那么可以考虑换个队排了。还有就是收银员的速度了，如果碰上了连 钱都点不清楚的新手，那就有的等了。另外，时机也很重要，可能 5 分钟前还人满为患的收款台，现在已是人去楼空，这时候交款可是很爽啊，当然，前提是那过去的 5 分钟里所做的事情比排队要有意义 (不过我还没发现什么事情比排队还无聊的)。
I/O 系统也和超市排队有很多类似之处:
r/s+w/s 类似于交款人的总数
平均队列长度(avgqu-sz)类似于单位时间里平均排队人的个数
平均服务时间(svctm)类似于收银员的收款速度
平均等待时间(await)类似于平均每人的等待时间
平均I/O数据(avgrq-sz)类似于平均每人所买的东西多少
I/O 操作率 (%util)类似于收款台前有人排队的时间比例。
我们可以根据这些数据分析出 I/O 请求的模式，以及 I/O 的速度和响应时间。
下面是别人写的这个参数输出的分析
# iostat -x 1avg-cpu: %user %nice %sys %idle16.24 0.00 4.31 79.44Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util/dev/cciss/c0d00.00 44.90 1.02 27.55 8.16 579.59 4.08 289.80 20.57 22.35 78.21 5.00 14.29上面的 iostat 输出表明秒有 28.57 次设备 I/O 操作: 总IO(io)/s = r/s(读) +w/s(写) = 1.02+27.55 = 28.57 (次/秒) 其中写操作占了主体 (w:r = 27:1)。
平均每次设备 I/O 操作只需要 5ms 就可以完成，但每个 I/O 请求却需要等上 78ms，为什么? 因为发出的 I/O 请求太多 (每秒钟约 29 个)，假设这些请求是同时发出的，那么平均等待时间可以这样计算:
平均等待时间 = 单个 I/O 服务时间 * ( 1 + 2 + … + 请求总数-1) / 请求总数
应用到上面的例子: 平均等待时间 = 5ms * (1+2+…+28)/29 = 70ms，和 iostat 给出的78ms 的平均等待时间很接近。这反过来表明 I/O 是同时发起的。
每秒发出的 I/O 请求很多 (约 29 个)，平均队列却不长 (只有 2 个 左右)，这表明这 29 个请求的到来并不均匀，大部分时间 I/O 是空闲的。
一秒中有 14.29% 的时间 I/O 队列中是有请求的，也就是说，85.71% 的时间里 I/O 系统无事可做，所有 29 个 I/O 请求都在142毫秒之内处理掉了。
delta(ruse+wuse)/delta(io) = await = 78.21 => delta(ruse+wuse)/s =78.21 * delta(io)/s = 78.21*28.57 = 2232.8，表明每秒内的I/O请求总共需要等待2232.8ms。所以平均队列长度应为 2232.8ms/1000ms = 2.23，而 iostat 给出的平均队列长度 (avgqu-sz) 却为 22.35，为什么?! 因为 iostat 中有 bug，avgqu-sz 值应为 2.23，而不是 22.35
参考
http://oplinux.com/order/iostat.html