UNIX Storage Diagnostics

run blk new|usr sys idl wai stl| used free buff cach| read writ| read writ5.0 0 0| 43 1 55 0 0|28.5G 7116M 7883M 18.4G|1.00 6.00 | 512B 29k5.0 0 0| 44 1 55 0 0|28.5G 7119M 7883M 18.4G| 0 0 | 0 0 5.0 0 0| 44 1 55 0 0|28.5G 7116M 7883M 18.4G|15.0 20.0 |1410k 154k5.0 0 0| 44 1 54 0 0|28.5G 7115M 7883M 18.4G|1.00 7.00 | 512B 37k5.0 0 0| 43 2 54 0 0|28.5G 7115M 7883M 18.4G| 0 2.00 | 0 548k5.0 0 0| 43 2 55 0 0|28.5G 7115M 7883M 18.4G|1.00 7.00 | 512B 64k5.0 0 0| 43 1 55 0 0|28.5G 7114M 7883M 18.4G|3.00 12.0 | 17k 73k5.0 0 0| 44 1 55 0 0|28.5G 7114M 7883M 18.4G| 0 0 | 0 0 5.0 0 0| 47 2 52 0 0|28.5G 7120M 7883M 18.4G|1.00 14.0 | 512B 189k5.0 0 0| 44 1 55 0 0|28.5G 7120M 7883M 18.4G| 0 8.00 | 0 88k6.0 0 0| 45 1 54 0 0|28.5G 7115M 7883M 18.4G|10.0 0 |1280k 0 5.0 0 0| 43 1 55 1 0|28.5G 7120M 7883M 18.4G|1.00 4.00 | 512B 21k6.0 0 0| 44 1 55 0 0|28.5G 7123M 7883M 18.4G| 0 0 | 0 0 5.0 0 0| 44 2 54 0 0|28.5G 7123M 7883M 18.4G|1.00 2.00 | 512B 4608B6.0 0 0| 44 1 55 0 0|28.5G 7123M 7883M 18.4G| 0 56.0 | 0 364k

-p Enable Process Stats

-c Enable CPU Stats

-m Enable Memory Stats

-r Enable I/O Stats

-d Enable Disk Stats

fio

Random Write Test

sudo fio --name=myjob \

--ioengine=libaio \

--readwrite=randwrite \

--size=5120M \

--numjobs=4 \

--runtime=240 \

--group_reporting

Arbitrary job name

Linux native asynchronous I/O

Random writes

Total size of file I/O for each thread of this job

How many clones of this job to run

How long job will run for (in seconds)

Displays aggregate totals for all jobs

iostat

sysstat

iostat -xcdmN 5

-x Extended statistics

-c CPU statistics

-d Device utilisation

-m Megabytes per second

-N Device mapper names

5 refresh every 5 seconds

For full details:man iostat

iostat -htx 1 1

-h Human Readable

-t Include time in output

-x Extended statistics

1 refresh every 1 second

1 for one iteration only

iotop

CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine SWAPIN and IO %

task_delayacct is set to off by default from Linux Kernel 5.14 onwards.uname --kernel-release

To dynamically set it to on...sudo sysctl kernel.task_delayacct=1
To disable...sudo sysctl kernel.task_delayacct=0
Note that turning this on may have some effect on system performance

The PRIO column shows the Thread priority...idle - "A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero." (1)be/0 - Best Effort priority 0be/4 - Best Effort priority 4rt/4 - Real Time priority 4Priority levels range from 0 to 7 with a lower number being higher priority.

blktrace

fatrace

sudo fatrace

inotify

iwatch

Bibliography

https://www.geeksforgeeks.org/dstat-command-in-linux-with-examples/https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-threadripper-pro-workstation/https://linoxide.com/linux-how-to/measure-disk-performance-fio/https://scoutapm.com/blog/understanding-disk-i-o-when-should-you-be-worriedhttps://unixutils.com/monitoring-disk-iops-in-linux/https://www.percona.com/blog/2014/06/25/why-util-number-from-iostat-is-meaningless-for-mysql-capacity-planning/https://unix.stackexchange.com/questions/60091/where-is-the-documentation-for-what-sda-sdb-dm-0-dm-1-meanhttps://unix.stackexchange.com/questions/86875/determining-specific-file-responsible-for-high-i-o
IOPShttps://unixutils.com/monitoring-disk-iops-in-linux/
iotophttps://superuser.com/questions/610581/iotop-complains-config-task-delay-acct-not-enabled-in-kernel-only-for-specifichttps://www.kernel.org/doc/html/latest/accounting/delay-accounting.html#usagehttps://www.mankier.com/8/iotop-chttps://unix.stackexchange.com/questions/238444/iotop-and-ionice-p-show-a-different-class-priority-for-the-same-process(1) https://linux.die.net/man/1/ionice