# twa, 3ware performance issue



## thedude (May 4, 2009)

Hey everybody, I just threw freebsd 7.1 on my server to replace Debian, and I have hit a snag...
The system is a dual quad core, currently only with 4GB of ram, and a 3ware 9650se-12 raid card with 7 750GB drives hooked to it in raid-6. With linux I could easily get around 100+MB/sec writing, but right now I'm only capable of getting around 2-3Mb/sec with freebsd. The array is partitioned with GPT with one partition using the whole 3.4TB and then i just ran newfs on the single partition.

I ran Bonnie++ and here are the results:

```
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.93d       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
me.c 8G   450  98  3110   0  3152   1   762  98 266188  54 374.0  12
Latency             36924us     735ms   12761ms   26482us     148ms   65865us
Version 1.93d       ------Sequential Create------ --------Random Create--------
me.com  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    26   0 +++++ +++    57   0    26   0 +++++ +++    50   0
Latency              2075ms      16us     491ms     460ms      29us    3713ms
1.93c,1.93d,me.com,1,1241508767,8G,,450,98,3110,0,3152,1,762,98,266188,54,374.0,12,16,,,,,26,0,+++++,+++,57,0,26,0,+++++,+++,50,0,36924us,735ms,12761ms,26482us,148ms,65865us,2075ms,16us,491ms,460ms,29us,3713ms
```

Right now it's running 7.1-Release and I just realized there are some patches i need to apply, but I doubt that's going to fix my problem

Here's the tw_cli output

```
### CLI Version:      x86_64 (64 bit)
### Time Stamp:       15:33.04 04-May-2009
### Host Name:        me.com
### OS Version:       FreeBSD 7.1-RELEASE
### Driver Version:   3.70.05.001
### Controller ID:    0
### Model:            9650SE-12ML
### Firmware:         FE9X 4.06.00.004
### BIOS:             BE9X 4.05.00.015
### Serial #:         L322605A8020287
### Available Memory: 224MB

==========================================================================
Diagnostic Information on Controller //me.com/c0 ...
--------------------------------------------------------------------------

No print log has been saved yet.
BBU status word : 0x0
```

If you have any advice I would appreciate it, I'm pulling my hair out right now

Thanks
James


----------



## thedude (May 4, 2009)

Correction from the speeds posted above, I meant to say 50MB+ instead 100MB+ for writing

- I ran `dd bs=1024 if=/dev/zero of=/file/on/disks count=1M` and the best speed I could get was 3MB/sec
- I just upgraded to 7.2-RELEASE and the problem is still the same. 
- Something I changed between the reformat, was that i installed a new nic, http://www.newegg.com/Product/Product.aspx?Item=N82E16833106123, which seems to be working fine...
- I have tried hw.pci.msi_enable=0 & hw.pci.msix_enable=0 & vfs.read_max=256 & vfs.hirunningspace=5MB but the results were the same...
- Also here is the dmesg output that I should have put in the first post. Also, 192.168.1.26 is the desktop that I am using to ssh into the server. I'm not really sure why there is the arp message at the bottom.
(in next message because it was too long..)


----------



## thedude (May 4, 2009)

```
FreeBSD 7.2-RELEASE #0: Fri May  1 07:18:07 UTC 2009
    root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5405  @ 2.00GHz (2000.01-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x10676  Stepping = 6
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xce33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA,<b19>>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 4
usable memory = 4281298944 (4082 MB)
avail memory  = 4108750848 (3918 MB)
ACPI APIC Table: <PTLTD          APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
3ware device driver for 9000 series storage controllers, version: 3.70.05.001
twa0: <3ware 9000 series Storage Controller> port 0x2000-0x20ff mem 0xd8000000-0xd9ffffff,0xda100000-0xda100fff irq 16 at device 0.0 on pci3
twa0: [ITHREAD]
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-12ML, 12 ports, Firmware FE9X 4.06.00.004, BIOS BE9X 4.05.00.015
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci4: <ACPI PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x3000-0x301f mem 0xda020000-0xda03ffff,0xda000000-0xda01ffff irq 18 at device 0.0 on pci4
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:30:48:35:95:02
em1: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x3020-0x303f mem 0xda060000-0xda07ffff,0xda040000-0xda05ffff irq 19 at device 0.1 on pci4
em1: Using MSI interrupt
em1: [FILTER]
em1: Ethernet address: 00:30:48:35:95:03
pcib5: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pci0: <base peripheral> at device 8.0 (no driver attached)
pcib8: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci8: <ACPI PCI bus> on pcib8
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 17 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 18 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xda600000-0xda6003ff irq 17 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb3: EHCI version 1.0
usb3: companion controllers, 2 ports each: usb0 usb1 usb2
usb3: <Intel 63XXESB USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3
uhub3: 6 ports with 6 removable, self powered
pcib9: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci9: <ACPI PCI bus> on pcib9
vgapci0: <VGA-compatible display> port 0x4000-0x40ff mem 0xd0000000-0xd7ffffff,0xda300000-0xda30ffff irq 18 at device 1.0 on pci9
em2: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x4400-0x443f mem 0xda340000-0xda35ffff,0xda320000-0xda33ffff irq 19 at device 3.0 on pci9
em2: [FILTER]
em2: Ethernet address: 00:1b:21:31:51:6d
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
atapci1: <Intel 63XXESB2 SATA300 controller> port 0x1890-0x1897,0x1884-0x1887,0x1888-0x188f,0x1880-0x1883,0x1870-0x187f mem 0xda600400-0xda6007ff irq 19 at device 31.2 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
plip0: <PLIP network interface> on ppbus0
plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
p4tcc3: <CPU Frequency Thermal Control> on cpu3
cpu4: <ACPI CPU> on acpi0
p4tcc4: <CPU Frequency Thermal Control> on cpu4
cpu5: <ACPI CPU> on acpi0
p4tcc5: <CPU Frequency Thermal Control> on cpu5
cpu6: <ACPI CPU> on acpi0
p4tcc6: <CPU Frequency Thermal Control> on cpu6
cpu7: <ACPI CPU> on acpi0
p4tcc7: <CPU Frequency Thermal Control> on cpu7
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xccfff,0xcd000-0xcdfff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ugen0: <American Power Conversion Back-UPS RS 1500 FW:8.g9 .D USB FW:g9, class 0/0, rev 1.10/1.06, addr 2> on uhub0
Timecounters tick every 1.000 msec
acd0: DVDR <HL-DT-STDVD-RAM GSA-H55N/1.03> at ata0-master UDMA66
ad4: 76319MB <Seagate ST380815AS 3.AAD> at ata2-master SATA150
GEOM_LABEL: Label for provider ad4s1a is ufsid/49fcfe3a86657425.
GEOM_LABEL: Label for provider ad4s1d is ufsid/49fcfe40873fc1d8.
GEOM_LABEL: Label for provider ad4s1e is ufsid/49fcfe3ae0162169.
GEOM_LABEL: Label for provider ad4s1f is ufsid/49fcfe3a82e6c7ad.
da0 at twa0 bus 0 target 0 lun 0
da0: <AMCC 9650SE-12M DISK 4.06> Fixed Direct Access SCSI-5 device 
da0: 100.000MB/s transfers
da0: 3576225MB (7324108800 512 byte sectors: 255H 63S/T 455904C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #7 Launched!
GEOM_LABEL: Label for provider da0 is ufsid/49fd82fc4db46291.
GEOM_LABEL: Label for provider da0p1 is ufsid/49fd888cf78698db.
GEOM_LABEL: Label for provider da0p1 is ufs/data.
Trying to mount root from ufs:/dev/ad4s1a
GEOM_LABEL: Label ufsid/49fcfe3a86657425 removed.
GEOM_LABEL: Label for provider ad4s1a is ufsid/49fcfe3a86657425.
GEOM_LABEL: Label ufsid/49fcfe3ae0162169 removed.
GEOM_LABEL: Label for provider ad4s1e is ufsid/49fcfe3ae0162169.
GEOM_LABEL: Label ufsid/49fcfe3a82e6c7ad removed.
GEOM_LABEL: Label for provider ad4s1f is ufsid/49fcfe3a82e6c7ad.
GEOM_LABEL: Label ufsid/49fcfe40873fc1d8 removed.
GEOM_LABEL: Label for provider ad4s1d is ufsid/49fcfe40873fc1d8.
GEOM_LABEL: Label ufsid/49fcfe3a86657425 removed.
GEOM_LABEL: Label ufsid/49fcfe3ae0162169 removed.
GEOM_LABEL: Label ufsid/49fcfe3a82e6c7ad removed.
GEOM_LABEL: Label ufsid/49fcfe40873fc1d8 removed.
GEOM_LABEL: Label ufsid/49fd82fc4db46291 removed.
GEOM_LABEL: Label ufs/data removed.
GEOM_LABEL: Label ufsid/49fd888cf78698db removed.
arp: 192.168.1.26 is on em0 but got reply from 00:1a:92:64:26:96 on em2
```

Again I could really use some help right now as I am completely stumped,
Thanks
James


----------



## Christopher (May 5, 2009)

Do you have write caching enabled on the array?  You can see this from the 3dm2 web management interface or from the tw_cli(8) utility.

For example:


```
//titan> /c0 show    

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-5    OK             -       -       256K    698.461   ON     OFF    

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     232.88 GB   488397168     WD-WCAL73146174     
p1     OK               u0     232.88 GB   488397168     WD-WCAL73172147     
p2     OK               u0     232.88 GB   488397168     WD-WCAL73288441     
p3     OK               u0     232.88 GB   488397168     WD-WCAL72055687     

Name  OnlineState  BBUReady  Status    Volt     Temp     Hours  LastCapTest
---------------------------------------------------------------------------
bbu   On           Yes       OK        OK       OK       255    26-Mar-2009
```


----------



## thedude (May 5, 2009)

I just enabled write cache the write cache and I'm now seeing write speeds over 150MB (when doing dd if=/dev/zero bs=1024 count=2M..) The thing is I do not have a BBU, just ordered it, so I had disabled this feature when debian was on the server and I never noticed any speed difference.
So I overlooked this setting as I was trying to figure out what was going on. The weird thing is that my write performance was fine on linux  without cache, so I'm not sure what's up with that.

Thanks for the help though


----------



## User23 (May 13, 2009)

Strange behavior. I using some 9550er 3ware Controllers and never noticed such problems. Did you contacted the 3ware Support too for this problem?


----------



## thedude (May 26, 2009)

No, I never contacted 3ware because I never really felt that there was an issue with the card, because it worked fine while using linux I figured there must be some configuration error on my part. Had I not been able to figure it out then I probably would have contacted them, but luckily I didn't have to go that far...

What I still think is strange is that with cache off while running Debian, the performance was still fairly good, but in FreeBSD, with the cache off, the performance was terrible...


----------



## User23 (Jul 27, 2009)

Now i got the same behaviour on a 7.2 release amd64 with a new 3ware 9550SXU-08LP / PCI-X / Single.

Without the write cache enabled the disk performance is really low. This do not depend on the raid level. Even if i use the disks as JBOD the perfomance is to low.

We using 3ware Controller for years on FreeBSD, i never recognized such a behaviour in the past.

Ill try to contact 3ware.

Any other FreeBSD Users can confirm that behaviour?

---
da1 and da2 are just single disks: 2x Hitachi HDT721010SLA360

tested with:

```
iozone -s 16g -r 8m -f /test/3ware/test.dat
        Iozone: Performance Test of File I/O
                Version $Revision: 3.319 $
                Compiled for 64 bit mode.
                Build: freebsd 

        Run began: Tue Aug  4 13:10:09 2009

        File size set to 16777216 KB
        Record Size 8192 KB
        Command line used: iozone -s 16g -r 8m -f /test/3ware/test.dat
        Output is in Kbytes/sec
        Time Resolution = 0.000003 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        16777216    8192
```

Without WriteCache enabled:

```
Disks   da0   da1   da2 pass0 pass1 pass2
KB/t   0.00 64.00 64.00  0.00  0.00  0.00
tps       0   112   113     0     0     0
MB/s   0.00  7.01  7.05  0.00  0.00  0.00
%busy     0   100   100     0     0     0
```

With WriteCache enabled:

```
Disks   da0   da1   da2 pass0 pass1 pass2
KB/t   0.00 64.00 64.00  0.00  0.00  0.00
tps       0  1507  1508     0     0     0
MB/s   0.00 94.18 94.26  0.00  0.00  0.00
%busy     0   100   100     0     0     0
```

Minimum throughput is more than 60 MB/s .


----------



## Fneufneu (Aug 25, 2009)

if you disable the write cache in your controller firmware, it will also disable the disk cache.
you can force it enable with tw_cli.

more info: 
http://www.3ware.com/KB/article.aspx?id=15555&cNode=5T3M5P


----------



## thedude (Aug 25, 2009)

interesting, thanks


----------

