# Horrible iSCSI (istgt) performance



## olav (Mar 19, 2011)

I get on average around 1MB/s on a gigabit network.

I've created a zfs volume which I've exported as a block device with istgt
[CMD=""]zfs create -V 330G -b 512 tank/vol[/CMD]

Jumbo frames are enabled. MTU is set to 7000. It also runs on a seperate network/vlan.
I've tried two different switches, both has same performance.

I guess it's a configuration issue?

/usr/local/etc/istgt/istgt.conf

```
17 [Global]
 18   Comment "Global section"
 19   # node name (not include optional part)
 20   NodeBase "iqn.backupbay.com"
 21
 22   # files
 23   PidFile /var/run/istgt.pid
 24   AuthFile /usr/local/etc/istgt/auth.conf
 25
 26   # directories
 27   # for removable media (virtual DVD/virtual Tape)
 28   MediaDirectory /var/istgt
 29
 30   # syslog facility
 31   LogFacility "local7"
 32
 33   # socket I/O timeout sec. (polling is infinity)
 34   Timeout 30
 35   # NOPIN sending interval sec.
 36   NopInInterval 20
 37
 38   # authentication information for discovery session
 39   DiscoveryAuthMethod Auto
 40   #DiscoveryAuthGroup AuthGroup9999
 41
 42   # reserved maximum connections and sessions
 43   # NOTE: iSCSI boot is 2 or more sessions required
 44   MaxSessions 32
 45   MaxConnections 8
 46
 47   # maximum number of sending R2T in each connection
 48   # actual number is limited to QueueDepth and MaxCmdSN and ExpCmdSN
 49   # 0=disabled, 1-256=improves large writing
 50   MaxR2T 128
 51
 52   # iSCSI initial parameters negotiate with initiators
 53   # NOTE: incorrect values might crash
 54   MaxOutstandingR2T 16
 55   DefaultTime2Wait 2
 56   DefaultTime2Retain 60
 57   FirstBurstLength 262144
 58   MaxBurstLength 1048576
 59   MaxRecvDataSegmentLength 262144
 60
 61   # NOTE: not supported
 62   InitialR2T Yes
 63   ImmediateData Yes
 64   DataPDUInOrder Yes
 65   DataSequenceInOrder Yes
 66   ErrorRecoveryLevel 0
 67
 68 [UnitControl]
 69   Comment "Internal Logical Unit Controller"
 70   #AuthMethod Auto
 71   AuthMethod CHAP Mutual
 72   AuthGroup AuthGroup1
 73   # this portal is only used as controller (by istgtcontrol)
 74   # if it's not necessary, no portal is valid
 75   #Portal UC1 [::1]:3261
 76   Portal UC1 127.0.0.1:3261
 77   # accept IP netmask
 78   #Netmask [::1]
 79   Netmask 127.0.0.1
 80
 81 # You should set IPs in /etc/rc.conf for physical I/F
 82 [PortalGroup1]
 83   Comment "SINGLE PORT TEST"
 84   # Portal Label(not used) IP(IPv6 or IPv4):Port
 85   #Portal DA1 [2001:03e0:06cf:0003:021b:21ff:fe04:f405]:3260
 86   Portal DA1 192.168.3.36:3260
 87
 96 [InitiatorGroup1]
 97   Comment "Initiator Group1"
 98   #InitiatorName "iqn.1991-05.com.microsoft:saturn"
 99   # special word "ALL" match all of initiators
100   InitiatorName "ALL"
101   Netmask 192.168.3.0/24
102
103 # TargetName, Mapping, UnitType, LUN0 are minimum required
104 [LogicalUnit1]
105   Comment "Hard Disk Sample"
106   # full specified iqn (same as below)
107   #TargetName iqn.2007-09.jp.ne.peach.istgt:disk1
108   # short specified non iqn (will add NodeBase)
109   TargetName disk1
110   TargetAlias "Data Disk1"
111   # use initiators in tag1 via portals in tag1
112   Mapping PortalGroup1 InitiatorGroup1
113   # accept both CHAP and None
114   AuthMethod Auto
115   AuthGroup AuthGroup2
116   #UseDigest Header Data
117   UseDigest Auto
118   UnitType Disk
119   # SCSI INQUIRY - Vendor(8) Product(16) Revision(4) Serial(16)
120   UnitInquiry "FreeBSD" "iSCSI Disk" "0123" "10000001"
121   # Queuing 0=disabled, 1-255=enabled with specified depth.
122   #QueueDepth 32
123
124   # override global setting if need
125   #MaxOutstandingR2T 16
126   #DefaultTime2Wait 2
127   #DefaultTime2Retain 60
128   #FirstBurstLength 262144
129   #MaxBurstLength 1048576
130   #MaxRecvDataSegmentLength 262144
131   #InitialR2T Yes
132   #ImmediateData Yes
133   #DataPDUInOrder Yes
134   #DataSequenceInOrder Yes
135   #ErrorRecoveryLevel 0
136
137   # LogicalVolume for this unit on LUN0
138   # for file extent
139   #LUN0 Storage /tank/iscsi/istgt-disk1 10GB
140   # for raw device extent
141   #LUN0 Storage /dev/ad4 Auto
142   # for ZFS volume extent
143   LUN0 Storage /dev/zvol/tank/vol Auto
```

I was hoping to get something around at least 50MB/s with iSCSI. The same zfs pool can easily read and write over 70-80MB/s with Samba (and no jumbo frames).


----------



## mamalos (Mar 19, 2011)

Try to isolate the problem. Run your experiment without the use of vlans or other virtual devices. Run *vmstat* and/or *iostat* on both machines and see if something peculiar is happening. Test it without jumbo frame support using the normal MTU (1500). Try exporting a ufs filesystem of the same physical disk. Crosscheck performance to see where the bottleneck comes from. Isolate your problem.

Check your logs!

By the way, have you checked your *dmesg* in case it shows something about sysctl tuning?


----------



## olav (Mar 20, 2011)

I've not tuned sysctl, I use default settings. None of the servers seems to have any io/cpu issues. I've tested without jumbo frames and got same result.

I'm gonna try to export an empty file as a volume from the ufs partition
Can't find anything in the logs either.

Interesting though, gstat says the iscsi device is working 100% when I use dd


----------



## olav (Mar 20, 2011)

Something is really wrong. After I rebooted the initiator now, I can't seem to connect to the iscsi target at all.

I find this error message on the target

```
Mar 20 11:54:53 zpool istgt[3262]: istgt_iscsi.c:4328:istgt_iscsi_send_nopin: ***ERROR*** before Full Feature
Mar 20 11:54:53 zpool istgt[3262]: istgt_iscsi.c:4817:worker: ***ERROR*** iscsi_send_nopin() failed
```

The initiator says

```
recvpdu: Socket is not connected
recvpdu failed
```


----------



## olav (Mar 24, 2011)

The extremely slow iSCSI was because of a broken network cable. I replaced it and now it runs very good! However I get maximum 50MB/s, the target is far from stressed. Are there other things I can do improve performance?


----------



## dennylin93 (Mar 24, 2011)

There are many ways of tuning ZFS. There isn't a best method as setups vary. Try searching the mailing lists since many people have posted tuning stuff (especially Jeremy Chadwick).


----------



## AndyUKG (Mar 24, 2011)

I think the first thing to do would be try without ZFS. I'd be interested to see what performance you get using a disk device directly, i.e. /dev/ada1 or what ever.

I also get terrible istgt performance with ZFS, but don't currently have the kit to do the testing I propose above as my istgt system is already in production.

Thanks, Andy.


----------



## olav (Mar 24, 2011)

I get 90-80MB/s with the Ubuntu initiator. I don't have a (fast) raw harddrive at the moment to test with.

I do have problems with my switch though, it doesn't seem to like jumbo frames. I need to test it more. A direct cable to each network card works fine with jumbo frames (but I don't get any performance benefits). But if anyone else is thinking about buying the NetGear GS724T, don't do it.


----------



## AndyUKG (Mar 25, 2011)

olav said:
			
		

> I don't have a (fast) raw harddrive at the moment to test with.



I think any disk would do to test, if the problem is just ZFS related you will see a huge jump in performance even with a slow disk.


----------



## kometen (Apr 12, 2011)

*iscsi performance*

Hi.

I've installed a server with zfs version 28 and FreeBSD 8.2 prerelease. The pool consists of 11 disks in raidz2. The disks are 2 TB sata disks on an areca-controller. The server has 12 GB of RAM and a xeon E5620 @ 2.4 Ghz. I installed net/istgt from ports. The only thing I tuned was 'QueueDepth 64'.

I have mounted the iscsi-disk from Windows 7/2008r2 and vmware 4.1 and have performance peaking at 118 MB/s which is very nice on plain GB nics. 


```
[LogicalUnit99]
  Comment "disk99"
  TargetName disk99
  TargetAlias "Data Disk99"
  Mapping PortalGroup1 InitiatorGroup1
  AuthMethod CHAP
  AuthGroup AuthGroup1
  UseDigest Auto
  UnitType Disk
  QueueDepth 64
  LUN0 Storage /dev/zvol/data/iscsi/disk99 Auto
```

Regards
Claus


----------



## AndyUKG (Apr 12, 2011)

Hi Claus,

So what has caused the jump in performance? Just the V28 ZFS you think?

Thanks Andy.


----------



## kometen (Apr 13, 2011)

*iscsi-performance*

Hi.

The only setting I changed was queue depth, which was commented out in the example iscsi.conf-file. I did not try zfs ver. 15 which is default in 8.2 but patched it to ver. 28 before creating the pool.

regards
Claus


----------



## AndyUKG (Apr 14, 2011)

Hi Claus,

  I don't see any noticeable difference when varying the size of queue depth. Do you actually see those speeds when moving real data around or are you seeing that from a windows benchmark utility? I see very good performance using benchmarking utils, but when actually moving real data the performance is very low...
If you really have got good performance then the only obvious explanation would be ZFS v28, I am using v15 currently,

thanks Andy.


----------



## kometen (Apr 15, 2011)

*iscsi-performance*

Hi.

I copied my Download-folder on Win 7 using both samba and iscsi, the smaller files don't utilize the network completely (of course) but the larger iso-files have no problem moving approx. to 100 MB/s, sometimes more, sometimes less.

The folder has 43.3 GB of data.

regards
Claus


----------



## peetaur (Jun 7, 2012)

The solution, as stated by a few, is to set QueueDepth (I found 64 works well, but only tested the VirtualBox initiator).

Why is this thread not marked solved? Is it because you wanted to know why the different initiators will perform well on the same default unset QueueDepth? Or did different initiators not benefit from QueueDepth? (I ask for my own personal knowledge, as I am starting to use istgt)


----------



## frijsdijk (Jun 11, 2012)

I too can get no faster than about 50MB/sec (in this case, a Windows host with iSCSI initiator, running Veeam backup software for VMware, that's writing to the disk that is connected via a 1Gbit link (dark fiber) to the FreeBSD host running istgt).

Iperf gets more traffic over the link (can max out the 1Gbit link), but istgt seems to get no faster than about half the 1Gbit link.

Probably because it's not running in the kernel but in user space?


----------



## peetaur (Jun 14, 2012)

I tested istgt with a file on zfs; it went only 4.5 MB/s! A zvol went over 100. Is this what you did? (and due to zvol hangs, I'm reverting to NFS for now.)


----------



## throAU (Jun 20, 2012)

As an aside, have you found any decent documentation for istgt?

I gave up with it because I couldn't make sense of the documentation.  I'm by no means a storage guru, but I had run an iSCSI SAN for 2-3 years at that point...


----------



## peetaur (Jun 20, 2012)

One online howto made istgt segfault.  But editing the sample config files worked fine, but no I found no documentation whatsoever.... just examples by 3rd parties. I wanted documentation so I would know what every setting meant, but found none.

In addition to the no documentation, also, if you typo in the istgt.conf file and try to reload/refresh the config, istgt can hang. So I don't think I want to use istgt at all. But there seems to be no good alternative. But if there is, I would love to hear about it.


----------



## phoenix (Jun 20, 2012)

Something to keep in mind when using NTFS-on-iSCSI-on-ZFS:  when you create the zvol, make sure that you match the recordsize to the NTFS block size, and (if possibile) set the MTU of the link to be the same.  If you don't, then you will be doing a lot of extra roundtrips for all reads and writes to the zvol.


----------



## peetaur (Jun 27, 2012)

peetaur said:
			
		

> I tested istgt with a file on zfs; it went only 4.5 MB/s! A zvol went over 100.



I tested this again, with VERY different results. 

Previously the initiator machine was using FreeBSD with VirtualBox's built in initiator, and 1Gbps. And as I said before, it was only around 4.5 MB/s (writing).

This time, I used a Linux machine, and it runs as fast as zvols (95-110 MB/s over 1Gbps, and 210-230 over 10 Gbps). And for comparison, netcat over the 10Gbps goes about 550 MB/s, and 2 threads goes 950 MB/s; I'm not sure what to do to speed it up for a single thread or iSCSI.

But this doesn't mean there is anything wrong with FreeBSD. It might simply be keeping better data integrity but Linux vbox doesn't. If I test this with Linux KVM, which I know does not have data integrity issues like vbox does, and get the slow result, I would blame zfs. If I get the fast result, I would blame FreeBSD or their changes to vbox.


----------



## dave (Jun 29, 2012)

Don't forget to add your NICs into the equation.  There are good NICs and bad.  Just because it says gigabit doesn't mean you will get the same from NIC to NIC.  Some rely on the CPU, some don't.


----------



## peetaur (Jun 29, 2012)

dave said:
			
		

> Don't forget to add your NICs into the equation.  There are good NICs and bad.  Just because it says gigabit doesn't mean you will get the same from NIC to NIC.  Some rely on the CPU, some don't.



If you are referring to my tests:

I doubt that a CPU or a cheap network card would be the difference between 4.5 MB/s and 40-80 MB/s. And the test clearly showed the zil at 100% load with the FreeBSD + vbox machine. And other tests (eg. scp) always show fast enough results.


----------



## dave (Jun 29, 2012)

peetaur said:
			
		

> If you are referring to my tests:
> 
> I doubt that a CPU or a cheap network card would be the difference between 4.5 MB/s and 40-80 MB/s. And the test clearly showed the zil at 100% load with the FreeBSD + vbox machine. And other tests (eg. scp) always show fast enough results.



I meant that more as a general reply because I noted that noone had mentioned it yet.  I was not referring specifically to your tests.


----------



## olav (Jun 30, 2012)

peetaur said:
			
		

> If you are referring to my tests:
> 
> I doubt that a CPU or a cheap network card would be the difference between 4.5 MB/s and 40-80 MB/s. And the test clearly showed the zil at 100% load with the FreeBSD + vbox machine. And other tests (eg. scp) always show fast enough results.



Actually, when it comes to iSCSI there is a HUGE difference between a good and a BAD NIC. I tested first with a Realtek NIC and then tried an Intel. I went from 5MB/s to 125MB/s.


----------



## tinusb (Aug 9, 2012)

Hey all! Is there any possibility you can post some of your istgt.conf files as example? I sit with a similar problem of bad performance, not a very powerful pc, but expect better performance than what I'm getting...


iostat reports about 8MB/sec stable throughput....


----------



## dave (Aug 9, 2012)

tinusb said:
			
		

> Hey all! Is there any possibility you can post some of your istgt.conf files as example? I sit with a similar problem of bad performance, not a very powerful pc, but expect better performance than what I'm getting...
> 
> iostat reports about 8MB/sec stable throughput....



When you say "not a very powerful pc" - what are we talking here?

Do you get different throughput using scp?

Is your CPU spiking when you do your tests?  If so, you may want to look for a better network card, one that doesn't have to interrupt the CPU.

Do your drives support the speed you are expecting?  What about the bus that the drives are connected to?


----------



## tinusb (Aug 10, 2012)

dave said:
			
		

> When you say "not a very powerful pc" - what are we talking here?
> 
> Do you get different throughput using scp?
> 
> ...



Hi,

Haven't checked using scp yet... 

SCP copy speed is:

2900kb/sec

No spiking of the CPU. Drives are SATA2, connected via a SATA1 bay, so can only do SATA1 - with normal on board controller.

Specs are: 
CPU AMD A8-3850 CPU (2900Mhz)
RAM 8GB Mushkin DDR-1333 
MOBO Gigabyte A75-D3H
HDD WD10EARX WD10EZRX WD10EZRX (thus 3x 1TB WD Green)
NIC Realtek 8168/8111 Gigabit; and INTeL/Pro1000 (Have the INTeL cards in our own machines, and do manage to get 85MB/sec tho)

When I take a look a the performance graphs in ESXi, I get a maximum throughput at times of about 300Mbps - utilizing a Gbit connection by only 30%? I do understand there are overheads, etc...

Regards
Tinus


----------



## dave (Aug 10, 2012)

olav said:
			
		

> Actually, when it comes to iSCSI there is a HUGE difference between a good and a BAD NIC. I tested first with a Realtek NIC and then tried an Intel. I went from 5MB/s to 125MB/s.



Is it worth a try to switch the NIC to intel?


----------



## olav (Aug 11, 2012)

Absolutely. I recommend the cheap Intel desktop ones

http://www.ebay.com/itm/Intel-Pro-1...US_Internal_Network_Cards&hash=item58953167c2

http://www.amazon.com/Intel-Gigabit...344680554&sr=1-10&keywords=intel+desktop+1000


----------



## tinusb (Aug 11, 2012)

olav said:
			
		

> Absolutely. I recommend the cheap Intel desktop ones
> 
> http://www.ebay.com/itm/Intel-Pro-1...US_Internal_Network_Cards&hash=item58953167c2
> 
> http://www.amazon.com/Intel-Gigabit...344680554&sr=1-10&keywords=intel+desktop+1000



Tried with INTeL cards, no difference. 

Own setup has got intel, with a Adaptec 51645 card with drives as JBOD - and it performs like crazy! But I can't believe that maximum I can get on the iSCSI is only about 3mb/sec?


----------



## peetaur (Aug 13, 2012)

Just to repeat the same thing again, you DID update QueueDepth to 64, right? Because out of anything I tested, these things are all that matters.

Here are some example snippets:


```
[Global]
...
# QueueDepth is limited by this number, and I don't know if it is per connection or accross all (due to zero documentation available) so I raised it a bunch.
MaxR2T 256
...

[LogicalUnit1]
  ...
  # 64 was perfectly sufficient for VirtualBox clients, but something else I tried ... maybe Proxmox, had errors until doubling it again.
  QueueDepth 128
  ...
```


----------



## larrypatrickmaloney (Aug 15, 2012)

I've gotta chime in on this...

So there are some 'alternate' ways to do iSCSI.

First of all make sure you've got your nic's setup with optimal MTU's (for FreeBSD best to set to 8244)

2nd of all, make sure you create your block devices with 4k or 8k alignment.

Now for the good stuff...

Run multiple istgt's on your target. 

The istgt target client is single threaded bound.

To get around this, launch multiple istgt, with each istgt providing a single block device.

Once you see the target devices on the initiator side, STRIPE ON THE INITIATOR. 

Try it.

Larry


----------

