Storage performance stats¶
| Postgres-OSM | AISB HDD | AISB SSD | VTTI Infra | |
|---|---|---|---|---|
| Read iops | 1065 | 89 | 1434 | 6421 |
| Read b/w (KiB/s) | 8500 | 710 | 11,470 | 51,384 |
| Write iops | 354 | 29 | 478 | 2140 |
| Write b/w (KiB/s) | 2830 | 238 | 3835 | 17,100 |
postgres-osm.vtti.vt.edu¶
The following are results from running fio on postgres-osm.vtti.vt.edu which is using Dell MD storage:
read-write.fio:
[read-write]
rw=rw
rwmixread=75
# 18g for 2+ drives
size=13g
directory=/data/md3220i/neal
fadvise_hint=0
blocksize=8k
direct=0
# 1 job per available CPU thread
numjobs=12
nrfiles=1
runtime=1h
time_based
exec_prerun=echo 3 > /proc/sys/vm/drop_caches
$ fio read-write.fio
Jobs: 12 (f=12): [M(12)][100.0%][r=85.1MiB/s,w=28.6MiB/s][r=10.9k,w=3658 IOPS][eta 00m:00s]
read-write: (groupid=0, jobs=1): err= 0: pid=620429: Tue Mar 29 13:42:08 2022
read: IOPS=1060, BW=8483KiB/s (8687kB/s)(29.1GiB/3600017msec)
clat (nsec): min=1762, max=664766k, avg=934269.12, stdev=6774057.88
lat (nsec): min=1803, max=664766k, avg=934426.42, stdev=6774053.92
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 17], 95.00th=[ 2704],
| 99.00th=[ 24249], 99.50th=[ 27395], 99.90th=[ 52167], 99.95th=[ 80217],
| 99.99th=[304088]
bw ( KiB/s): min= 256, max=1396848, per=8.30%, avg=8491.45, stdev=19281.12, samples=7192
iops : min= 32, max=174608, avg=1061.40, stdev=2410.15, samples=7192
write: IOPS=353, BW=2828KiB/s (2896kB/s)(9944MiB/3600017msec); 0 zone resets
clat (usec): min=4, max=3482, avg=16.34, stdev=17.76
lat (usec): min=4, max=3482, avg=16.65, stdev=17.94
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 40],
| 99.00th=[ 94], 99.50th=[ 125], 99.90th=[ 190], 99.95th=[ 233],
| 99.99th=[ 347]
bw ( KiB/s): min= 48, max=463520, per=8.30%, avg=2831.10, stdev=6415.61, samples=7192
iops : min= 6, max=57940, avg=353.87, stdev=801.95, samples=7192
lat (usec) : 2=0.01%, 4=35.42%, 10=40.16%, 20=13.34%, 50=5.17%
lat (usec) : 100=1.10%, 250=0.65%, 500=0.06%, 750=0.03%, 1000=0.04%
lat (msec) : 2=0.14%, 4=0.33%, 10=1.01%, 20=1.18%, 50=1.27%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.39%, sys=2.14%, ctx=211975, majf=0, minf=41
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3817455,1272776,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-write: (groupid=0, jobs=1): err= 0: pid=620430: Tue Mar 29 13:42:08 2022
read: IOPS=1063, BW=8509KiB/s (8713kB/s)(29.2GiB/3600006msec)
clat (nsec): min=1788, max=650652k, avg=931337.50, stdev=6736900.29
lat (nsec): min=1833, max=650652k, avg=931494.37, stdev=6736897.27
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 16], 95.00th=[ 2540],
| 99.00th=[ 24511], 99.50th=[ 27395], 99.90th=[ 51643], 99.95th=[ 79168],
| 99.99th=[299893]
bw ( KiB/s): min= 256, max=1237328, per=8.32%, avg=8516.88, stdev=21398.66, samples=7192
iops : min= 32, max=154666, avg=1064.57, stdev=2674.83, samples=7192
write: IOPS=354, BW=2837KiB/s (2905kB/s)(9974MiB/3600006msec); 0 zone resets
clat (usec): min=4, max=19131, avg=16.39, stdev=24.97
lat (usec): min=4, max=19131, avg=16.72, stdev=25.09
clat percentiles (usec):
| 1.00th=[ 7], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 40],
| 99.00th=[ 94], 99.50th=[ 126], 99.90th=[ 194], 99.95th=[ 235],
| 99.99th=[ 347]
bw ( KiB/s): min= 16, max=408272, per=8.32%, avg=2839.71, stdev=7093.68, samples=7192
iops : min= 2, max=51034, avg=354.95, stdev=886.71, samples=7192
lat (usec) : 2=0.01%, 4=36.05%, 10=39.55%, 20=13.40%, 50=5.13%
lat (usec) : 100=1.08%, 250=0.65%, 500=0.06%, 750=0.03%, 1000=0.04%
lat (msec) : 2=0.15%, 4=0.35%, 10=1.00%, 20=1.10%, 50=1.31%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.39%, sys=2.14%, ctx=212664, majf=0, minf=22
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3828847,1276629,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-write: (groupid=0, jobs=1): err= 0: pid=620431: Tue Mar 29 13:42:08 2022
read: IOPS=1063, BW=8510KiB/s (8714kB/s)(29.2GiB/3600025msec)
clat (nsec): min=1863, max=652679k, avg=930820.74, stdev=6724168.30
lat (nsec): min=1902, max=652679k, avg=930978.48, stdev=6724164.90
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 16], 95.00th=[ 2573],
| 99.00th=[ 24511], 99.50th=[ 27395], 99.90th=[ 51643], 99.95th=[ 78119],
| 99.99th=[299893]
bw ( KiB/s): min= 256, max=1145888, per=8.33%, avg=8521.60, stdev=20020.62, samples=7189
iops : min= 32, max=143234, avg=1065.16, stdev=2502.56, samples=7189
write: IOPS=354, BW=2839KiB/s (2907kB/s)(9980MiB/3600025msec); 0 zone resets
clat (usec): min=3, max=2356, avg=16.47, stdev=17.77
lat (usec): min=3, max=2356, avg=16.81, stdev=17.98
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 40],
| 99.00th=[ 95], 99.50th=[ 127], 99.90th=[ 194], 99.95th=[ 235],
| 99.99th=[ 338]
bw ( KiB/s): min= 80, max=382192, per=8.33%, avg=2842.57, stdev=6684.86, samples=7189
iops : min= 10, max=47774, avg=355.30, stdev=835.61, samples=7189
lat (usec) : 2=0.02%, 4=37.04%, 10=38.36%, 20=13.49%, 50=5.18%
lat (usec) : 100=1.12%, 250=0.66%, 500=0.07%, 750=0.03%, 1000=0.04%
lat (msec) : 2=0.14%, 4=0.36%, 10=0.99%, 20=1.14%, 50=1.29%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.38%, sys=2.16%, ctx=212268, majf=0, minf=53
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3829391,1277390,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-write: (groupid=0, jobs=1): err= 0: pid=620432: Tue Mar 29 13:42:08 2022
read: IOPS=1067, BW=8540KiB/s (8745kB/s)(29.3GiB/3600191msec)
clat (nsec): min=1851, max=617466k, avg=927710.06, stdev=6680427.01
lat (nsec): min=1892, max=617467k, avg=927867.08, stdev=6680424.36
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 16], 95.00th=[ 2671],
| 99.00th=[ 24249], 99.50th=[ 27395], 99.90th=[ 50594], 99.95th=[ 77071],
| 99.99th=[299893]
bw ( KiB/s): min= 32, max=1105024, per=8.36%, avg=8552.03, stdev=20990.95, samples=7189
iops : min= 4, max=138128, avg=1068.96, stdev=2623.87, samples=7189
write: IOPS=355, BW=2843KiB/s (2911kB/s)(9995MiB/3600191msec); 0 zone resets
clat (usec): min=3, max=831, avg=16.49, stdev=17.58
lat (usec): min=4, max=832, avg=16.84, stdev=17.75
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 40],
| 99.00th=[ 95], 99.50th=[ 126], 99.90th=[ 192], 99.95th=[ 233],
| 99.99th=[ 338]
bw ( KiB/s): min= 128, max=364416, per=8.35%, avg=2847.33, stdev=6960.37, samples=7188
iops : min= 16, max=45552, avg=355.90, stdev=870.05, samples=7188
lat (usec) : 2=0.01%, 4=36.01%, 10=39.43%, 20=13.40%, 50=5.22%
lat (usec) : 100=1.13%, 250=0.65%, 500=0.06%, 750=0.03%, 1000=0.04%
lat (msec) : 2=0.14%, 4=0.34%, 10=1.01%, 20=1.15%, 50=1.28%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.39%, sys=2.16%, ctx=213097, majf=0, minf=32
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3843103,1279365,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-write: (groupid=0, jobs=1): err= 0: pid=620433: Tue Mar 29 13:42:08 2022
read: IOPS=1069, BW=8552KiB/s (8758kB/s)(29.4GiB/3600020msec)
clat (nsec): min=1723, max=650784k, avg=926506.76, stdev=6714713.61
lat (nsec): min=1764, max=650784k, avg=926662.89, stdev=6714710.66
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 16], 95.00th=[ 2507],
| 99.00th=[ 24511], 99.50th=[ 27395], 99.90th=[ 50594], 99.95th=[ 77071],
| 99.99th=[304088]
bw ( KiB/s): min= 256, max=1493680, per=8.36%, avg=8560.84, stdev=24173.30, samples=7192
iops : min= 32, max=186710, avg=1070.07, stdev=3021.66, samples=7192
write: IOPS=356, BW=2850KiB/s (2918kB/s)(9.78GiB/3600020msec); 0 zone resets
clat (usec): min=4, max=4970, avg=16.43, stdev=18.61
lat (usec): min=4, max=4971, avg=16.77, stdev=18.79
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 40],
| 99.00th=[ 95], 99.50th=[ 126], 99.90th=[ 194], 99.95th=[ 235],
| 99.99th=[ 355]
bw ( KiB/s): min= 16, max=497616, per=8.36%, avg=2852.78, stdev=8050.80, samples=7192
iops : min= 2, max=62202, avg=356.58, stdev=1006.35, samples=7192
lat (usec) : 2=0.01%, 4=36.11%, 10=39.31%, 20=13.49%, 50=5.17%
lat (usec) : 100=1.11%, 250=0.65%, 500=0.07%, 750=0.03%, 1000=0.04%
lat (msec) : 2=0.15%, 4=0.36%, 10=0.98%, 20=1.16%, 50=1.28%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.40%, sys=2.15%, ctx=213442, majf=0, minf=79
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3848607,1282505,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-write: (groupid=0, jobs=1): err= 0: pid=620434: Tue Mar 29 13:42:08 2022
read: IOPS=1075, BW=8606KiB/s (8812kB/s)(29.5GiB/3600009msec)
clat (nsec): min=1841, max=639862k, avg=921304.25, stdev=6612868.88
lat (nsec): min=1883, max=639863k, avg=921459.80, stdev=6612866.16
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 16], 95.00th=[ 2507],
| 99.00th=[ 24249], 99.50th=[ 27395], 99.90th=[ 50594], 99.95th=[ 74974],
| 99.99th=[295699]
bw ( KiB/s): min= 256, max=1250480, per=8.41%, avg=8609.43, stdev=21917.59, samples=7196
iops : min= 32, max=156310, avg=1076.14, stdev=2739.70, samples=7196
write: IOPS=358, BW=2870KiB/s (2939kB/s)(9.85GiB/3600009msec); 0 zone resets
clat (usec): min=3, max=984, avg=16.33, stdev=17.44
lat (usec): min=3, max=984, avg=16.66, stdev=17.62
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 39],
| 99.00th=[ 94], 99.50th=[ 126], 99.90th=[ 194], 99.95th=[ 235],
| 99.99th=[ 330]
bw ( KiB/s): min= 80, max=417584, per=8.42%, avg=2871.46, stdev=7282.48, samples=7196
iops : min= 10, max=52198, avg=358.91, stdev=910.31, samples=7196
lat (usec) : 2=0.02%, 4=36.77%, 10=38.84%, 20=13.41%, 50=5.11%
lat (usec) : 100=1.07%, 250=0.65%, 500=0.06%, 750=0.03%, 1000=0.04%
lat (msec) : 2=0.15%, 4=0.35%, 10=1.01%, 20=1.11%, 50=1.30%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.39%, sys=2.16%, ctx=214044, majf=0, minf=22
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3872607,1291622,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-write: (groupid=0, jobs=1): err= 0: pid=620435: Tue Mar 29 13:42:08 2022
read: IOPS=1063, BW=8509KiB/s (8713kB/s)(29.2GiB/3600001msec)
clat (nsec): min=1860, max=656209k, avg=931927.50, stdev=6750216.00
lat (nsec): min=1902, max=656209k, avg=932083.70, stdev=6750213.57
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 16], 95.00th=[ 2573],
| 99.00th=[ 24249], 99.50th=[ 27395], 99.90th=[ 51119], 99.95th=[ 79168],
| 99.99th=[304088]
bw ( KiB/s): min= 512, max=1348864, per=8.32%, avg=8513.41, stdev=20392.90, samples=7195
iops : min= 64, max=168608, avg=1064.15, stdev=2549.11, samples=7195
write: IOPS=354, BW=2833KiB/s (2901kB/s)(9959MiB/3600001msec); 0 zone resets
clat (usec): min=3, max=2108, avg=16.42, stdev=17.82
lat (usec): min=4, max=2108, avg=16.73, stdev=17.98
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 40],
| 99.00th=[ 95], 99.50th=[ 127], 99.90th=[ 196], 99.95th=[ 233],
| 99.99th=[ 338]
bw ( KiB/s): min= 128, max=447600, per=8.31%, avg=2834.24, stdev=6791.91, samples=7195
iops : min= 16, max=55950, avg=354.27, stdev=848.99, samples=7195
lat (usec) : 2=0.01%, 4=35.85%, 10=39.67%, 20=13.41%, 50=5.13%
lat (usec) : 100=1.13%, 250=0.66%, 500=0.06%, 750=0.03%, 1000=0.04%
lat (msec) : 2=0.15%, 4=0.35%, 10=0.99%, 20=1.17%, 50=1.28%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.39%, sys=2.15%, ctx=212476, majf=0, minf=43
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3829087,1274783,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-write: (groupid=0, jobs=1): err= 0: pid=620436: Tue Mar 29 13:42:08 2022
read: IOPS=1066, BW=8529KiB/s (8734kB/s)(29.3GiB/3600189msec)
clat (nsec): min=1800, max=667536k, avg=928683.26, stdev=6678413.73
lat (nsec): min=1842, max=667536k, avg=928840.10, stdev=6678410.46
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 16], 95.00th=[ 2540],
| 99.00th=[ 24511], 99.50th=[ 27395], 99.90th=[ 51119], 99.95th=[ 76022],
| 99.99th=[299893]
bw ( KiB/s): min= 256, max=1140336, per=8.35%, avg=8541.27, stdev=20174.23, samples=7189
iops : min= 32, max=142542, avg=1067.62, stdev=2521.78, samples=7189
write: IOPS=355, BW=2842KiB/s (2910kB/s)(9993MiB/3600189msec); 0 zone resets
clat (usec): min=4, max=1055, avg=16.37, stdev=17.53
lat (usec): min=4, max=1055, avg=16.71, stdev=17.71
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 40],
| 99.00th=[ 94], 99.50th=[ 126], 99.90th=[ 192], 99.95th=[ 233],
| 99.99th=[ 347]
bw ( KiB/s): min= 48, max=380416, per=8.34%, avg=2846.38, stdev=6724.04, samples=7189
iops : min= 6, max=47552, avg=355.78, stdev=840.51, samples=7189
lat (usec) : 2=0.02%, 4=38.56%, 10=36.92%, 20=13.46%, 50=5.13%
lat (usec) : 100=1.11%, 250=0.66%, 500=0.07%, 750=0.03%, 1000=0.04%
lat (msec) : 2=0.15%, 4=0.35%, 10=0.99%, 20=1.15%, 50=1.28%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.39%, sys=2.15%, ctx=212516, majf=0, minf=69
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3838191,1279089,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-write: (groupid=0, jobs=1): err= 0: pid=620437: Tue Mar 29 13:42:08 2022
read: IOPS=1068, BW=8545KiB/s (8750kB/s)(29.3GiB/3600038msec)
clat (nsec): min=1843, max=626975k, avg=927113.65, stdev=6694809.99
lat (nsec): min=1890, max=626975k, avg=927269.96, stdev=6694807.05
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 16], 95.00th=[ 2474],
| 99.00th=[ 24511], 99.50th=[ 27395], 99.90th=[ 51119], 99.95th=[ 76022],
| 99.99th=[304088]
bw ( KiB/s): min= 224, max=1305504, per=8.36%, avg=8557.36, stdev=21710.06, samples=7189
iops : min= 28, max=163188, avg=1069.63, stdev=2713.76, samples=7189
write: IOPS=356, BW=2849KiB/s (2918kB/s)(9.78GiB/3600038msec); 0 zone resets
clat (usec): min=3, max=827, avg=16.20, stdev=17.41
lat (usec): min=4, max=827, avg=16.53, stdev=17.59
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8],
| 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 39],
| 99.00th=[ 94], 99.50th=[ 126], 99.90th=[ 192], 99.95th=[ 229],
| 99.99th=[ 330]
bw ( KiB/s): min= 96, max=437104, per=8.36%, avg=2853.38, stdev=7256.10, samples=7189
iops : min= 12, max=54638, avg=356.65, stdev=907.01, samples=7189
lat (usec) : 2=0.02%, 4=37.89%, 10=38.06%, 20=13.14%, 50=5.03%
lat (usec) : 100=1.08%, 250=0.65%, 500=0.06%, 750=0.03%, 1000=0.05%
lat (msec) : 2=0.15%, 4=0.36%, 10=0.96%, 20=1.17%, 50=1.28%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.39%, sys=2.14%, ctx=213318, majf=0, minf=60
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3845407,1282227,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-write: (groupid=0, jobs=1): err= 0: pid=620438: Tue Mar 29 13:42:08 2022
read: IOPS=1070, BW=8563KiB/s (8769kB/s)(29.4GiB/3600002msec)
clat (nsec): min=1840, max=687127k, avg=924869.97, stdev=6645338.83
lat (nsec): min=1881, max=687148k, avg=925026.37, stdev=6645335.93
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 16], 95.00th=[ 2507],
| 99.00th=[ 24249], 99.50th=[ 27395], 99.90th=[ 51119], 99.95th=[ 76022],
| 99.99th=[299893]
bw ( KiB/s): min= 256, max=835968, per=8.38%, avg=8575.16, stdev=18453.46, samples=7189
iops : min= 32, max=104496, avg=1071.86, stdev=2306.68, samples=7189
write: IOPS=356, BW=2854KiB/s (2923kB/s)(9.80GiB/3600002msec); 0 zone resets
clat (usec): min=3, max=4017, avg=16.42, stdev=17.90
lat (usec): min=4, max=4017, avg=16.76, stdev=18.08
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 40],
| 99.00th=[ 94], 99.50th=[ 124], 99.90th=[ 192], 99.95th=[ 235],
| 99.99th=[ 351]
bw ( KiB/s): min= 128, max=284560, per=8.38%, avg=2858.32, stdev=6162.27, samples=7189
iops : min= 16, max=35570, avg=357.28, stdev=770.29, samples=7189
lat (usec) : 2=0.02%, 4=37.59%, 10=37.87%, 20=13.41%, 50=5.22%
lat (usec) : 100=1.11%, 250=0.64%, 500=0.06%, 750=0.03%, 1000=0.04%
lat (msec) : 2=0.15%, 4=0.35%, 10=1.01%, 20=1.13%, 50=1.27%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.39%, sys=2.15%, ctx=213552, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3853375,1284438,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-write: (groupid=0, jobs=1): err= 0: pid=620439: Tue Mar 29 13:42:08 2022
read: IOPS=1064, BW=8517KiB/s (8721kB/s)(29.2GiB/3600004msec)
clat (nsec): min=1844, max=638124k, avg=930850.40, stdev=6712148.20
lat (nsec): min=1889, max=638124k, avg=931007.92, stdev=6712146.01
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 16], 95.00th=[ 2704],
| 99.00th=[ 24249], 99.50th=[ 27395], 99.90th=[ 51643], 99.95th=[ 78119],
| 99.99th=[308282]
bw ( KiB/s): min= 256, max=1311152, per=8.33%, avg=8521.40, stdev=18994.48, samples=7195
iops : min= 32, max=163894, avg=1065.14, stdev=2374.31, samples=7195
write: IOPS=355, BW=2841KiB/s (2909kB/s)(9986MiB/3600004msec); 0 zone resets
clat (usec): min=4, max=2228, avg=16.30, stdev=17.46
lat (usec): min=4, max=2228, avg=16.64, stdev=17.65
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 40],
| 99.00th=[ 94], 99.50th=[ 125], 99.90th=[ 190], 99.95th=[ 227],
| 99.99th=[ 322]
bw ( KiB/s): min= 48, max=435744, per=8.33%, avg=2842.08, stdev=6303.88, samples=7195
iops : min= 6, max=54468, avg=355.24, stdev=787.99, samples=7195
lat (usec) : 2=0.01%, 4=37.46%, 10=38.16%, 20=13.39%, 50=5.08%
lat (usec) : 100=1.09%, 250=0.65%, 500=0.06%, 750=0.03%, 1000=0.04%
lat (msec) : 2=0.14%, 4=0.34%, 10=1.00%, 20=1.15%, 50=1.30%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.39%, sys=2.15%, ctx=212783, majf=0, minf=38
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3832479,1278231,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-write: (groupid=0, jobs=1): err= 0: pid=620440: Tue Mar 29 13:42:08 2022
read: IOPS=1061, BW=8489KiB/s (8693kB/s)(29.1GiB/3600014msec)
clat (nsec): min=1864, max=653776k, avg=933716.34, stdev=6779886.02
lat (nsec): min=1905, max=653776k, avg=933873.35, stdev=6779882.46
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 16], 95.00th=[ 2606],
| 99.00th=[ 24249], 99.50th=[ 27395], 99.90th=[ 52167], 99.95th=[ 80217],
| 99.99th=[304088]
bw ( KiB/s): min= 256, max=1071456, per=8.30%, avg=8494.75, stdev=18037.68, samples=7194
iops : min= 32, max=133932, avg=1061.81, stdev=2254.71, samples=7194
write: IOPS=353, BW=2830KiB/s (2898kB/s)(9951MiB/3600014msec); 0 zone resets
clat (usec): min=4, max=3553, avg=16.42, stdev=17.88
lat (usec): min=4, max=3554, avg=16.75, stdev=18.07
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 41],
| 99.00th=[ 94], 99.50th=[ 125], 99.90th=[ 188], 99.95th=[ 231],
| 99.99th=[ 343]
bw ( KiB/s): min= 64, max=350064, per=8.30%, avg=2832.35, stdev=5959.37, samples=7194
iops : min= 8, max=43758, avg=354.03, stdev=744.92, samples=7194
lat (usec) : 2=0.01%, 4=37.34%, 10=38.21%, 20=13.27%, 50=5.25%
lat (usec) : 100=1.12%, 250=0.65%, 500=0.06%, 750=0.03%, 1000=0.04%
lat (msec) : 2=0.15%, 4=0.35%, 10=1.00%, 20=1.11%, 50=1.32%
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.39%, sys=2.15%, ctx=212071, majf=0, minf=73
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3819967,1273676,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=99.9MiB/s (105MB/s), 8483KiB/s-8606KiB/s (8687kB/s-8812kB/s), io=351GiB (377GB), run=3600001-3600191msec
WRITE: bw=33.3MiB/s (34.9MB/s), 2828KiB/s-2870KiB/s (2896kB/s-2939kB/s), io=117GiB (126GB), run=3600001-3600191msec
Disk stats (read/write):
dm-6: ios=2731243/108914, merge=10/686, ticks=80548574/61183339, in_queue=134350816, util=100.00%, aggrios=341455/13614, aggrmerge=0/0, aggrticks=10058024/7642611, aggrin_queue=16759877, aggrutil=97.96%
sdf: ios=607788/27209, merge=0/0, ticks=19256046/15500961, in_queue=32908448, util=97.63%
sdd: ios=743853/27284, merge=0/0, ticks=20731856/15175241, in_queue=33908012, util=97.83%
sdb: ios=750764/27167, merge=0/0, ticks=20822979/15095075, in_queue=33792052, util=97.91%
sdi: ios=9/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdg: ios=9/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sde: ios=9/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdc: ios=9/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdh: ios=629201/27257, merge=0/0, ticks=19653313/15369611, in_queue=33470508, util=97.96%
```
{ data-search-exclude }
## Kubernetes Storage
Testing with [kubestr](https://github.com/kastenhq/kubestr) and the following:
```conf
[global]
name=fio-rand-RW
filename=fio-rand-RW
rw=randrw
rwmixread=75
bs=8K
direct=0
numjobs=4
time_based
runtime=900
[file1]
size=13G
ioengine=libaio
iodepth=16
AISB HDD (prod kubernetes)¶
$ kubestr fio -s ceph-block --fiofile kubestr-fio.fio
PVC created kubestr-fio-pvc-rjwvr
Pod created kubestr-fio-pod-rr425
Running FIO test (kubestr-fio.fio) on StorageClass (aisb-hdd) with a PVC of Size (100Gi)
Elapsed time- 16m42.74985945s
FIO test results:
FIO version - fio-3.20
Global options - ioengine= verify= direct=0 gtod_reduce=
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=89.158218 BW(KiB/s)=713
iops: min=16 max=142 avg=89.292709
bw(KiB/s): min=128 max=1136 avg=714.351685
write:
IOPS=29.727182 BW(KiB/s)=237
iops: min=4 max=64 avg=29.772398
bw(KiB/s): min=32 max=512 avg=238.190872
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=88.716888 BW(KiB/s)=709
iops: min=20 max=132 avg=88.858017
bw(KiB/s): min=160 max=1056 avg=710.874146
write:
IOPS=29.757481 BW(KiB/s)=238
iops: min=2 max=60 avg=29.807905
bw(KiB/s): min=16 max=480 avg=238.474945
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=88.823555 BW(KiB/s)=710
iops: min=22 max=150 avg=88.966034
bw(KiB/s): min=176 max=1200 avg=711.738281
write:
IOPS=29.714148 BW(KiB/s)=237
iops: min=6 max=58 avg=29.757795
bw(KiB/s): min=48 max=464 avg=238.074051
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=88.791237 BW(KiB/s)=710
iops: min=22 max=146 avg=88.922607
bw(KiB/s): min=176 max=1168 avg=711.390869
write:
IOPS=29.815224 BW(KiB/s)=238
iops: min=6 max=60 avg=29.868040
bw(KiB/s): min=48 max=480 avg=238.956009
Disk stats (read/write):
rbd3: ios=291111/106824 merge=0/0 ticks=2001468/1265872 in_queue=3266224, util=100.000000%
- OK
AISB SSD (prod kubernetes)¶
$ kubestr fio -s ceph-block --fiofile kubestr-fio.fio
PVC created kubestr-fio-pvc-tvm7m
Pod created kubestr-fio-pod-ql6rz
Running FIO test (kubestr-fio.fio) on StorageClass (aisb-ssd) with a PVC of Size (100Gi)
Elapsed time- 15m41.786659576s
FIO test results:
FIO version - fio-3.20
Global options - ioengine= verify= direct=0 gtod_reduce=
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=1434.051758 BW(KiB/s)=11472
iops: min=618 max=46274 avg=2311.910400
bw(KiB/s): min=4944 max=370192 avg=18495.287109
write:
IOPS=478.776123 BW(KiB/s)=3830
iops: min=218 max=15318 avg=771.827942
bw(KiB/s): min=1744 max=122544 avg=6174.628906
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=1433.734009 BW(KiB/s)=11469
iops: min=646 max=46058 avg=2311.435547
bw(KiB/s): min=5168 max=368464 avg=18491.501953
write:
IOPS=479.161682 BW(KiB/s)=3833
iops: min=216 max=15686 avg=772.469543
bw(KiB/s): min=1728 max=125488 avg=6179.761719
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=1434.169556 BW(KiB/s)=11473
iops: min=588 max=46230 avg=2312.064453
bw(KiB/s): min=4704 max=369840 avg=18496.521484
write:
IOPS=478.619476 BW(KiB/s)=3828
iops: min=206 max=15176 avg=771.632629
bw(KiB/s): min=1648 max=121408 avg=6173.066406
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=1434.560669 BW(KiB/s)=11476
iops: min=644 max=45376 avg=2312.718750
bw(KiB/s): min=5152 max=363008 avg=18501.738281
write:
IOPS=478.173920 BW(KiB/s)=3825
iops: min=192 max=15090 avg=770.878113
bw(KiB/s): min=1536 max=120704 avg=6167.017090
Disk stats (read/write):
rbd3: ios=1312775/1670600 merge=0/239 ticks=1025448/9430556 in_queue=10462112, util=98.853188%
- OK
ceph-block (infra-vtti kubernetes)¶
$ kubestr fio -s ceph-block --fiofile kubestr-fio.fio
PVC created kubestr-fio-pvc-vwvfk
Pod created kubestr-fio-pod-bfkk9
Running FIO test (kubestr-fio.fio) on StorageClass (ceph-block) with a PVC of Size (100Gi)
Elapsed time- 15m19.232006164s
FIO test results:
FIO version - fio-3.30
Global options - ioengine= verify= direct=0 gtod_reduce=
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=6421.946289 BW(KiB/s)=51375
iops: min=1042 max=188026 avg=9233.274414
bw(KiB/s): min=8336 max=1504208 avg=73866.671875
write:
IOPS=2141.833252 BW(KiB/s)=17134
iops: min=318 max=62806 avg=3079.395264
bw(KiB/s): min=2544 max=502448 avg=24635.556641
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=6423.115234 BW(KiB/s)=51384
iops: min=1062 max=186823 avg=9243.259766
bw(KiB/s): min=8496 max=1494589 avg=73946.546875
write:
IOPS=2141.602051 BW(KiB/s)=17132
iops: min=346 max=61839 avg=3081.812744
bw(KiB/s): min=2768 max=494717 avg=24654.919922
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=6416.462402 BW(KiB/s)=51331
iops: min=1038 max=185476 avg=9227.865234
bw(KiB/s): min=8304 max=1483815 avg=73823.406250
write:
IOPS=2139.217529 BW(KiB/s)=17113
iops: min=316 max=61917 avg=3076.374512
bw(KiB/s): min=2528 max=495342 avg=24611.474609
JobName:
blocksize= filesize=13G iodepth=16 rw=
read:
IOPS=6416.925293 BW(KiB/s)=51335
iops: min=1090 max=172306 avg=9232.954102
bw(KiB/s): min=8720 max=1378452 avg=73864.054688
write:
IOPS=2135.434326 BW(KiB/s)=17083
iops: min=338 max=57745 avg=3072.569336
bw(KiB/s): min=2704 max=461963 avg=24581.003906
Disk stats (read/write):
rbd2: ios=3352497/4673714 merge=0/37964 ticks=2246547/39291747 in_queue=30023592, util=97.921684%