# I/O streams capped at 128KB/t?



## t0ken (Jun 14, 2018)

I'm trying to come up with a solution to an issue I'm seeing on FreeBSD VMs installed as qemu/KVM guests (I'm sure the fact that they're guests have nothing to do with my issue, but I thought I'd mention it anyway).

From within a VM, I seem to be hitting a ~128KB/s limit on I/O streams, that only seems to be present in FreeBSD.  The first output below is to the local disk (local to the VM) on which the OS is installed, you can see that the KB/t (kilobytes per transfer) never go above ~128KB/t, though the IOPS (iostat tps field) go up as far as allowed for the VM. I ran a simple 'dd' with 'bs=1M count=1024' (columns are KB/t,tps,MB/s):


  0.00   0  0.00   0.00   0  0.00
 126.90 322 39.91   0.00   0  0.00
 127.59 1094 136.31   0.00   0  0.00
 126.79 1089 134.88   0.00   0  0.00
 127.85 1314 164.03   0.00   0  0.00
 127.57 1340 166.91   0.00   0  0.00
 127.56 1322 164.69   0.00   0  0.00
 127.64 1331 165.87   0.00   0  0.00
 126.87 425 52.71   0.00   0  0.00
  7.00   1  0.01   0.00   0  0.00
  0.00   0  0.00   0.00   0  0.00


The second disk I added (which is an RBD network disk, though I don't think that matters, thought I'd mention it) has it's IOPS limited to 75 (with some bursting allowed).  You can see that the KB/t still remains below 128KB/t, which creates a bottleneck as the IOPS cannot increase for this disk, giving it a very low throughput of ~9MB/s, which is horrible write speed for my case.


  0.00   0  0.00  124.69  75  9.17   0.00   0  0.00   0.00   0  0.00
  0.00   0  0.00  124.86  74  9.05   0.00   0  0.00   0.00   0  0.00
  0.00   0  0.00  124.64  75  9.19   0.00   0  0.00   0.00   0  0.00
  0.00   0  0.00  124.81  75  9.09   0.00   0  0.00   0.00   0  0.00
  0.00   0  0.00  124.61  76  9.23   0.00   0  0.00   0.00   0  0.00
  0.00   0  0.00  124.80  74  9.04   0.00   0  0.00   0.00   0  0.00
  0.00   0  0.00  124.77  75  9.14   0.00   0  0.00   0.00   0  0.00
  0.00   0  0.00  124.56  76  9.22   0.00   0  0.00   0.00   0  0.00
  0.00   0  0.00  123.24  75  9.03   0.00   0  0.00   0.00   0  0.00
  0.00   0  0.00  124.97  75  9.15   0.00   0  0.00   0.00   0  0.00
  4.00   1  0.00  124.73  75  9.13   0.00   0  0.00   0.00   0  0.00
  0.00   0  0.00  124.73  75  9.14   0.00   0  0.00   0.00   0  0.00


From within any number of Linux VMs (I've tried Ubuntu, Arch, and Fedora) on the same platform, using the RBD disk I do not have the same issue with speed, as those systems seem to scale up their data per second -- discounting the fact that the IOPS in this case burst up a bit to 118tps (this is from a VM running Arch Linux; columns are tps,KB_read/s,KB_wrtn/s,KB_read,KB_wrtn):

sdb               4.00         8.00        28.00          8         28
sdb              26.00         4.00     23820.00          4      23820
sdb              31.00         0.00     31488.00          0      31488
sdb             118.00         0.00    120832.00          0     120832
sdb              90.10         0.00     93275.25          0      94208
sdb               0.00         0.00         0.00          0          0



My question is, am I able to increase the KB/t from 128KB/t?  Or is this simply a limit of the FreeBSD kernel?  I'd prefer to be able to increase the KB/t (from within the VM) rather than IOPS (which I can do from the host system) if at all possible.

Thanks in advance for any insight.


----------

