# Marvell 88SE6145 with ZFS Timeouts



## aronjd (Dec 21, 2010)

Hi There,

I'm quite new with FreeBSD (I'm Good with Juniper ) 
Normally I do everything with Linux (Fedora or Ubuntu)
I have made a home server with 15T of storage.

The Hardware consists of a Jetway MB with a extra 4 ports SATA Daughter board.

MB: Jetway NC9C-550-LF (link)
Add-on: ADPE4S-LF(link)
Storage HDD: WD Caviar GREEN 2TB (internal 5disks)
OS DISK: Hitachi 160GB
Rest of the 15TB are USB disks (legacy)

The Controller on the board is a Marvell 88SE6145 Chip.
The controller on the MB is Intel.

The problem is as follows:

When I create the ZFS Pool with:

`zpool create tank ada0 ada1 ada3 ada4 ada5` (these are all internal disks)
ada2 is the OS disk

The tank is created when I run a `zpool status` i get the following error:


```
Dec 20 14:45:26 store kernel: ahcich2: Timeout on slot 10 Dec 20 14:45:26 store kernel: ahcich2: is 00000000 cs 00003000 ss 00003c00 rs 
00003c00 tfd 50040040 serr 00000000
```

Sometimes it says ahcich4 or some other number.

Sometimes a zpool status will go correct and the timeout occurs at a test (example: dd created test file)

This was with a 8.1-RELEASE version.

I just updated (today with cvsup nl mirror) the OS with 9.0-CURRENT. To see if there where any changes. But the problem still existed.

I detached some of the drives for the Controller and placed them on the internal MB controller. Everything works FINE.

But the strangest thing..... My / and SWAP drive are still connected on the Raid controller. Only the ZFS drives are on the Onboard controller.

I tested the / drive with some commands : find / -name (somequery) or portsnap fetch and update. some ports make.... Everything works flawlessly?

Just with ZFS it will go terrible wrong.

My question for you  What kind of info would you like? And is there a developer who loves a good challenge  
The Server is at the moment in test. So destroying data or reinstalling something is NO problem! when everything works fine the server will get a clean install and build.

uname -sr says: FreeBSD 9.0-Current

The /boot/loader file:


```
zfs_load="YES"
ahci_load="YES"
vfs.zfs.prefetch_disable=0      # enable prefetch
vfs.zfs.arc_max=134217728       # 128 MB
vm.kmem_size=536870912          # 512 MB
vm.kmem_size_max=536870912      # 512 MB
vfs.zfs.vdev.cache.size=8388608 #   8 MB
```


```
store# pciconf -lbcv  //removed USB, VGA, Serial and network
hostb0@pci0:0:0:0:      class=0x060000 card=0xa0108086 chip=0xa0108086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = HOST-PCI
    cap 09[e0] = vendor (length 8) Intel cap 5 version 1
pcib1@pci0:0:28:0:      class=0x060400 card=0x27d08086 chip=0x27d08086 rev=0x02 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801G (ICH7 Family) PCIe Root Port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 1 root port max data 128(128) link x0(x1)
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge card=0x27d08086
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
ecap 0002[100] = VC 1 max VC1
ecap 0005[180] = unknown 1
pcib2@pci0:0:28:1:      class=0x060400 card=0x27d28086 chip=0x27d28086 rev=0x02 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801G (ICH7 Family) PCIe Root Port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 1 root port max data 128(128) link x1(x1)
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge card=0x27d28086
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
ecap 0002[100] = VC 1 max VC1
ecap 0005[180] = unknown 1
pcib3@pci0:0:28:2:      class=0x060400 card=0x27d48086 chip=0x27d48086 rev=0x02 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801G (ICH7 Family) PCIe Root Port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 1 root port max data 128(128) link x1(x1)
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge card=0x27d48086
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
ecap 0002[100] = VC 1 max VC1
ecap 0005[180] = unknown 1
pcib4@pci0:0:28:3:      class=0x060400 card=0x27d68086 chip=0x27d68086 rev=0x02 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801G (ICH7 Family) PCIe Root Port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 1 root port max data 128(128) link x1(x1)
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge card=0x27d68086
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
ecap 0002[100] = VC 1 max VC1
ecap 0005[180] = unknown 1
pcib5@pci0:0:30:0:      class=0x060401 card=0x24488086 chip=0x24488086 rev=0xe2 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801 Family (ICH2/3/4/5/6/7/8/9-M) Hub Interface to PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
    cap 0d[50] = PCI Bridge card=0x24488086
isab0@pci0:0:31:0:      class=0x060100 card=0x27bc8086 chip=0x27bc8086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = PCI-ISA
    cap 09[e0] = vendor (length 12) Intel cap 1 version 0
                 features: Quick Resume, SATA RAID-5, Mobile, 6 PCI-e x1 slots, SATA RAID-0/1/10
ahci2@pci0:0:31:2:      class=0x010601 card=0x27c18086 chip=0x27c18086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801GB I/O Controller Hub SATA cc=AHCI'
    class      = mass storage
    subclass   = SATA
    bar   [10] = type I/O Port, range 32, base 0x9800, size  8, enabled
    bar   [14] = type I/O Port, range 32, base 0xa080, size  4, enabled
    bar   [18] = type I/O Port, range 32, base 0xa000, size  8, enabled
    bar   [1c] = type I/O Port, range 32, base 0x9c00, size  4, enabled
    bar   [20] = type I/O Port, range 32, base 0x9880, size 32, enabled
    bar   [24] = type Memory, range 32, base 0xfe9f3800, size 1024, enabled
    cap 05[80] = MSI supports 1 message enabled with 1 message
    cap 01[70] = powerspec 2  supports D0 D3  current D0
    cap 12[a8] = SATA Index-Data Pair
ahci0@pci0:4:0:0:       class=0x010601 card=0x2363197b chip=0x2363197b rev=0x02 hdr=0x00
    vendor     = 'JMicron Technology Corp.'
    device     = 'JMicron JMB362/JMB363 AHCI Controller (JMB36X)'
    class      = mass storage
    subclass   = SATA
    bar   [24] = type Memory, range 32, base 0xfeafe000, size 8192, enabled
    cap 01[68] = powerspec 2  supports D0 D3  current D0
    cap 10[50] = PCI-Express 1 legacy endpoint IRQ 2 max data 128(128) link x1(x1)
atapci0@pci0:4:0:1:     class=0x010185 card=0x2363197b chip=0x2363197b rev=0x02 hdr=0x00
    vendor     = 'JMicron Technology Corp.'
    device     = 'JMicron JMB362/JMB363 AHCI Controller (JMB36X)'
    class      = mass storage
    subclass   = ATA
    bar   [10] = type I/O Port, range 32, base 0xdc00, size  8, enabled
    bar   [14] = type I/O Port, range 32, base 0xd880, size  4, enabled
    bar   [18] = type I/O Port, range 32, base 0xd800, size  8, enabled
    bar   [1c] = type I/O Port, range 32, base 0xd480, size  4, enabled
    bar   [20] = type I/O Port, range 32, base 0xd400, size 16, enabled
    cap 01[68] = powerspec 2  supports D0 D3  current D0
pcib6@pci0:5:7:0:       class=0x060400 card=0x00000000 chip=0x221111ab rev=0x01 hdr=0x01
    vendor     = 'Marvell Semiconductor (Was: Galileo Technology Ltd)'
    class      = bridge
    subclass   = PCI-PCI
    cap 01[40] = powerspec 2  supports D0 D1 D2 D3  current D0
    cap 10[48] = PCI-Express 1 PCI to PCIe bridge max data 128(128) link x1(x1)
atapci1@pci0:6:0:0:     class=0x010400 card=0x614511ab chip=0x614511ab rev=0xa1 hdr=0x00
    vendor     = 'Marvell Semiconductor (Was: Galileo Technology Ltd)'
    device     = 'Add-on IC to provide 4x SATA Ports, attached to ICH7 (SthBridge?) via PCI-Express. (88SE6145)'
    class      = mass storage
    subclass   = RAID
    bar   [10] = type I/O Port, range 32, base 0xe480, size  8, enabled
    bar   [14] = type I/O Port, range 32, base 0xe400, size  4, enabled
    bar   [18] = type I/O Port, range 32, base 0xec00, size  8, enabled
    bar   [1c] = type I/O Port, range 32, base 0xe880, size  4, enabled
    bar   [20] = type I/O Port, range 32, base 0xe800, size 16, enabled
    bar   [24] = type Memory, range 32, base 0xfebbfc00, size 1024, enabled
    cap 01[48] = powerspec 2  supports D0 D1 D3  current D0
    cap 05[50] = MSI supports 1 message
    cap 10[e0] = PCI-Express 1 legacy endpoint max data 128(128) link x1(x1)
```


----------



## AndyUKG (Dec 22, 2010)

The errors would seem to indicate a hardware problem communicating with the drives. Why don't you do some more testing with the drives attached to the RAID controller to rule out ZFS as causing the problem and prove whether you have a general hardware issue? Ie format the drives with UFS and copy some data onto them, and/or dd/tar data to the disk devices. I'd recommend to try and access multiples drives simultaneously if doing one test a time doesn't show up any issues, obviously this will be happening when you are running them under ZFS.

thanks Andy.


----------



## aronjd (Dec 22, 2010)

Hi Andy,

As stated in my previous posts:

- When i connected the Drives to the internal controller everything works fine.
- My OS disk is connected to the Raidcontroller i updated my OS (make world) on it installed samba and made many [CMD="find /-name"]find /-name[/CMD] query's on them.

So, ZFS works fine when the drives affected with the problem are connected to the internal controller.
with other words the drives are OK.
My OS disk is connected to the Raidcontroller and the controller works fine.
So the Raidcontroller works fine. 

The only reason the system hangs is with ZFS on the raidcontroller.

With kind regards,

Aron Dijkstra


----------



## AndyUKG (Dec 22, 2010)

You have proved that the RAID controller works reliably with one disk attached (to one port or did you test all the ports?). My suggestion is to test it again with multiple disks attached, as this is what has caused you the errors (these are not ZFS errors).

Andy.


----------



## mav@ (Dec 23, 2010)

Just to note, I've ordered some board with Marvell 88SE6145 chip. May be in few weeks I can say something.


----------



## aronjd (Dec 25, 2010)

Hi,

I just reformatted the server. reinstalled it with the DVD (installsource is the main FreeBSD FTP server). now i'm back to 8.1-Release.
Now i only have the ahci driver in de boot.conf.

I connected 4 drives to the raid controller. I mounted them on my system (UFS2)
i DD't  a 1GB file on them. (i have issued the commands independently from each other)
It worked perfectly.

Now as we speak I'm putting a 50000GB file on every disk all dd commands are running together.
I have No errors so far!!

No errors ether in the messages log.

i will report back is the test succeeded. and When it does. i will try again with a raidz configuration.

G's

Aron


----------



## aronjd (Dec 25, 2010)

Back Again,

The Disks and controller handeld it correctly. There where NO time-outs.
Tomorrow i will create the ZFS pool.

G's

Aron


----------



## aronjd (Dec 27, 2010)

I'm back 

Sorry for the late reply.

I have created the zpool consisting of the 5 disks. the creation works fine.
But the strange thing is i could create the 1GB dd file and afterward the 10GB dd file.

The only difference is that i did not load all the zfs configuration options in the boot configuration load file.
But when i wanted to scrub the pool i got 4 timeouts.

I was wondering... Could it be the load on the controller? I have found this website:

http://butcher.heavennet.ru/patches/kernel/ata/marvell/

It states that the controller could be unstable on high loads.

What is your opinion about this? 

G's

Aron


----------



## danbi (Dec 28, 2010)

Just a note --- using four drives with separate filesystems is not the same as using four drives with one (complex) filesystem such as ZFS. ZFS will put much more stress on your hardware and if it does not behave properly (or the driver does not handle it well) you will see problems --- that problems you will almost never see in other 'light use' cases.

ZFS is not to blame here -- it works perfectly on properly working hardware, even if the hardware is 'low-end' or whatever.

As this is separate controlller.. does the chip get very hot? If so, it may help if you put a heatsink on it... or direct airflow that way somehow, maybe additional fan to blow at the controller (chips).


----------



## AndyUKG (Dec 28, 2010)

aronjd said:
			
		

> What is your opinion about this?



Well doesn't sound good. It's been suggested to others with stability issues to try limiting the port speed to 1.5Gb/sec via the sysctl tunable "hint.ahcich.X.sata_rev=1" where X is the controller (set in loader.conf). Aside from that if you aren't in a hurry you can wait and see if mav@ finds and fixes any issues with the driver. If you are in a hurry you could consider buying a PCI card based on a Sil3124 chip (I have several of these and they have been working well with ZFS).

ta Andy.


----------



## aronjd (Dec 29, 2010)

Hi There again,

I will put the controllers in a 1,5Gb mode. and hope that mav@ can find something
The speed is not important right now. For the next month or maybe 2 a speed of 10MB/sec is enough.

Thanks for the help!!!

Aron


----------



## danbi (Dec 30, 2010)

Don't worry so much about speed. It is very unlikely your drives can do more than 150MB/sec each.


----------



## aronjd (Dec 30, 2010)

To bad, That did not work..
It's as u said Danbi... The drives could not reach the 150MB/s So whats the point of clipping them on a speed the cannot reach..
At the first try after reboot the timeout reappeared while creating the Pool.

Is there a setting to clip the speed completely back to let's say 12MB/s ?

Thanks!

Aron Dijkstra


----------



## aronjd (Jan 4, 2011)

@ mav@: Did u recieve your controller? Hope that u find a solution 

@ anybody, Nobody an idea to clip the speed? I tried to make a ZFS Raidz with only 4 disks. and when there was a solution expand the Raidz with the 5the disk.. but ZFS does not support this?


----------



## mav@ (Jan 4, 2011)

aronjd said:
			
		

> @ mav@: Did u recieve your controller? Hope that u find a solution



Received the board today. Will try to allocate time for it.



			
				aronjd said:
			
		

> @ anybody, Nobody an idea to clip the speed?



SATA has no speeds less then 150MB/s.


----------



## aronjd (Jan 16, 2011)

Hi Mav@,

I'm back again 

I was wondering if you had any luck allocating time for the board  Or maybe can I help you in any way? (logging or testing). I'm no programmer, but maybe some other skills I can help with?

With kind regards,

Aron


----------



## ers (Jan 18, 2011)

Hi 

To aronjd - I had similar problems but no one caould help back then. Please verify that those timeouts shows not only under zfs. At first I was sure that this is ZFS but after this I found out that under UFS with very high load (especially under multiple threads) timeuts shows again.

Look at thread "zfs waits for [zio->io_cv] or [zfs]", maybe you find some similarities. Please inform me if yours tests gone the same way.

mav@ said back then he does not have enough documentation. After this I found that this controller is rather buggy and under linux could be helpful switching off NCQ, but I have no time and skills to test this fast enough ... ):

Despite AndyUKG doubts there was no problem with disks, cables, port mutipliers. The same disks with the same cales were used on another controller on the same board and they are stable (under ZFS and UFS). No port multipliers used.

To mav@ - I hope you will find soon what causes the problem since you have this controller.
*I want to offer you my help in this case *

Please inform us what is the status and how we can help you.

Best regards
ers


----------



## mav@ (Feb 12, 2011)

At last I've got time to test it closer. I can acknowledge that copying large file on ZFS of 4 disks causes command timeout just in few seconds. Same as with provided log messages, I suppose it is not a driver, but a controller problem, as after 30 seconds controller still reports that commands are not completed.

I've found that disabling NCQ helps with the problem. Together with one more workaround I've committed it to 9-CURRENT at SVN rev 218596: http://svn.freebsd.org/changeset/base/218596.

Update your sources please and try again.


----------



## monstrfolk (Feb 12, 2011)

This fix does not work with stable branch for FreeBSD 8. It fails because cam_ccb.h does not contain hba_vendor, hba_device, hba_subvendor, and hba_subdevice. I commented out lines 2669 - 2672 in ahci.c. The module now compiles. Question...Will it work for 8 by commenting out those lines or does hba_vendor, hba_device, hba_subvendor, and hba_subdevice have to be set?


----------



## mav@ (Feb 13, 2011)

Those fields are not existed and not needed at 8-STABLE. I would try to apply mentioned patch exactly http://svn.freebsd.org/viewvc/base/...view=patch&r1=218605&r2=218595&pathrev=218596, but removing those hba_XXX lines from 9-CURRENT file should probably also work.


----------



## monstrfolk (Feb 13, 2011)

The patch mav@ has posted works! I transferred 20gigs of files from drives connected to a Marvell 88se6145 controller. A few files were as large as 5gigs. 

Great job mav and thankyou!


----------



## monstrfolk (Feb 14, 2011)

Conducted more tests. Transfered 400gigs from a mirror zfs to a raidz2 on the Marvell 88se6145 Sata controller.

Sometimes the copying would take longer than I expected on some files, but always continued.


----------



## aronjd (Feb 14, 2011)

Hi  Everybody,

I hope i don't sound to much as a noob.. But can somebody explain to me how to insert this patch in FreeBSD stable (version 8.1 if it is possible)

I tried to install version 9 but it is WAY to buggy for my production environment.

Thanks!!


----------



## ers (Feb 15, 2011)

mav@ said:
			
		

> Those fields are not existed and not needed at 8-STABLE. I would try to apply mentioned patch exactly http://svn.freebsd.org/viewvc/base/...view=patch&r1=218605&r2=218595&pathrev=218596, but removing those hba_XXX lines from 9-CURRENT file should probably also work.



I can confirm that removing those hba_XXX lines from 9-CURRENT is working for 8.1-RELEASE, at least for now... 

to aronjd
At first I was trying to apply this patch to 8.1-RELEASE but ahci.c is much different...


```
Patching file ahci.c using Plan A...
Hunk #1 succeeded at 114.
Hunk #2 succeeded at 165 with fuzz 1 (offset -6 lines).
Hunk #3 failed at 1907.
1 out of 3 hunks failed--saving rejects to ahci.c.rej
done
```

Then I get ahci.c from repository and removed lines poited by mav@ (compiler complaints about not known fields)


```
cpi->hba_vendor = pci_get_vendor(parent);
cpi->hba_device = pci_get_device(parent);
cpi->hba_subvendor = pci_get_subvendor(parent);
cpi->hba_subdevice = pci_get_subdevice(parent);
```

and rebuild the kernel (http://www.freebsd.org/doc/handbook/kernelconfig-building.html)

Since then I am testing the new kernel and it appears to be stable (20GB in multiple threads so far), more tests soon...

*Excellent work - thank you mav@ !!! *


----------



## aronjd (Feb 15, 2011)

Everything sounds VERY promising, I will try to compile it in the kernel.

I have one latest question. How can I only download the ahci.c file from the repository? Last time I used cvsup everything got updated


----------



## ers (Feb 15, 2011)

Simple Stupid Method 

Use this link: http://svn.freebsd.org/viewvc/base/head/sys/dev/ahci/ahci.c
which contain log list of ahci.c commits.

From most top revision use link "view" to access ahci.c
Copy ahci.c and replace old one.
Correct it as decribed above.
Compile  & be happy


----------



## aronjd (Feb 15, 2011)

Hi Ers,

I have done everything, the kernel is building as we speak!
I keep my fingers crossed 

Thanks!


----------



## aronjd (Feb 16, 2011)

it works!!! GREAT!!!

Thanks mav and everybody else who helped!

btw. There is another fix that had been submitted after the marvell fix that one will not compile on 8.1
just a note to other people 

This post can be marked as solved


----------



## ers (Feb 17, 2011)

aronjd said:
			
		

> btw. There is another fix that had been submitted after the marvell fix that one will not compile on 8.1



Another fix?
Please give us a hint or a link


----------



## aronjd (Feb 18, 2011)

Well it's not a fix. just a caution 

If you use Revision 218605, That one will not compile into the kernel. Only the revision for Marvell will work. 
BTW, everything is still stable after 3 days of use 

Only thing is I cannot use *atacontrol* for spindown... It does not see any disks. I have to use *camcontrol* for it.


----------



## ers (Feb 18, 2011)

aronjd said:
			
		

> Well it's not a fix. just a caution
> 
> If you use Revision 218605, That one will not compile into the kernel. Only the revision for Marvell will work.
> BTW, everything is still stable after 3 days of use
> ...



I do not chatch...
mav@ submited *rev.218596* in which he accidentally dropped DH89xxCC/Patsburg chip IDs
in next *rev.218605* described _"Restore DH89xxCC/Patsburg chip IDs accentally dropped at r218596."_ he just restore them...
so *both of them should compile*.
I have compiled *rev.218605* into *8.1-RELEASE* with modifications mav@ described without any problems and everything is stable so far.

About atacontrol and camcontrol... are they dependent on enabling new ahci driver?


----------



## disi (Jul 13, 2011)

Anybody knows, if this patch got merged yet? I might have a similiar issue with stable 8.2 and cannot see anything regarding ahci got patched if I sync.


----------



## mav@ (Jul 13, 2011)

It is merged to 8-STABLE.


----------



## disi (Jul 16, 2011)

mav@ said:
			
		

> It is merged to 8-STABLE.



All upgraded and up and running, I currently run a backup job onto those drives...


----------

