Добрый день! Подскажите, пож-та, куда копать запись на rbd - минимум 1076MB/s,

а запись на cephfs на тех же osd - жалкие 22.1MB/s.
?

один файл или много мелких?

fio -ioengine=libaio -direct=1 -name=test -bs=4k -iodepth=16 -rw=write -randrepeat=0 --size=10G --directory /mnt/testing/ test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=23.1MiB/s][r=0,w=5916 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=46135: Fri Jan 28 12:44:09 2022 write: IOPS=5406, BW=21.1MiB/s (22.1MB/s)(10.0GiB/484852msec) slat (usec): min=4, max=3894, avg=29.63, stdev=16.53 clat (usec): min=997, max=209733, avg=2928.01, stdev=1660.17 lat (usec): min=1016, max=209785, avg=2957.91, stdev=1660.35 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1893], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2540], 50.00th=[ 2704], 60.00th=[ 2900], | 70.00th=[ 3097], 80.00th=[ 3458], 90.00th=[ 4047], 95.00th=[ 4686], | 99.00th=[ 6128], 99.50th=[ 6783], 99.90th=[ 9634], 99.95th=[11076], | 99.99th=[31851] bw ( KiB/s): min= 9304, max=27648, per=99.98%, avg=21622.56, stdev=2327.46, samples=969 iops : min= 2326, max= 6912, avg=5405.60, stdev=581.87, samples=969 lat (usec) : 1000=0.01% lat (msec) : 2=8.62%, 4=80.75%, 10=10.54%, 20=0.07%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.29%, sys=13.28%, ctx=2043845, majf=0, minf=212 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=21.1MiB/s (22.1MB/s), 21.1MiB/s-21.1MiB/s (22.1MB/s-22.1MB/s), io=10.0GiB (10.7GB), run=484852-484852msec

Внезапно, rbd_cache vs честный I/O

а чем смонтировано?

mount -t ceph xx.xx.xx.xx:6789,xx.xx.xx.xx:6789,xx.xx.xx.xx:6789:/ /mnt/testing -o mds_namespace=cephfs,name=admin,secret=

22 метра конечно совсем дно

А это ничего что он RBD тестит блоком 4M а cepfs блоком 4K ? 🙂

Уже ж разобрались :)

А блин)))