# BCM5720 Status



## chodong (May 2, 2012)

Is there an update on the driver for the BCM5720?
We just got a R720 from Dell and need to get the driver update to run FreeBSD 9.0.


----------



## Terry_Kennedy (May 3, 2012)

chodong said:
			
		

> Is there an update on the driver for the BCM5720?
> We just got a R720 from Dell and need to get the driver update to run FreeBSD 9.0.


If you update your kernel sources with net/cvsup or csup(1), rebuld the kernel and reboot, does it work?

If not, post the output from `# pciconf -l | grep 14e4`. (14e4 is Broadcom's PCI vendor ID.) The bge(4) maintainer, yongari@, does seem to follow this forum and can probably help.


----------



## chodong (May 3, 2012)

Thank you Terry for the reply.

Initially the four NIC ports were not recognized with the release of FreeBSD 9.0 so I simply got the latest copies of the bge driver and made a kernel.

This kernel was able to detect the BCM5720 NIC's four ports but I got NMI errors when I tried to ifconfig the ports.

I can certainly understand that there may be other files required. 

I will update my kernel sources and try again.


----------



## linuxunix (May 4, 2012)

Chodong,

Are you successful in making the driver work? Can you share the steps you carried out for making that driver work?


----------



## chodong (May 5, 2012)

Still no luck.

All four NIC ports are detected at boot time but I am unable to [font="Courier New"]ifconfig[/font] any port.

I get the following error messages:


```
NMI ... going to debugger[/font] ->repeated four times
kernel: NMI ISA 20, EISA ff
```

I did the following:


```
# svn checkout svn://svn.freebsd.org/base/stale/9/ /usr/src
# cd /usr/src
# make buildworld
# make buildkernel KERNCONF=GENERIC
# cd release
# make release
```
I then burnt a CD and booted/installed from it.

The following is the output of
[cmd=]# pciconf -l | grep 14e4[/cmd]


```
bge0@pci0:2:0:0      class=0x020000 card=0x1f5b1028 chip=0x165f14e4 rev=0x00 hdr=0x00
bge1@pci0:2:0:1      class=0x020000 card=0x1f5b1028 chip=0x165f14e4 rev=0x00 hdr=0x00
bge2@pci0:1:0:0      class=0x020000 card=0x1f5b1028 chip=0x165f14e4 rev=0x00 hdr=0x00
bge3@pci0:1:0:1      class=0x020000 card=0x1f5b1028 chip=0x165f14e4 rev=0x00 hdr=0x00
```

I don't know what I did wrong presuming the BCM5720 driver works.

If anyone has any ideas please let me know.


----------



## chodong (May 6, 2012)

I got his reply to a query on the Broadcom web site.




> Support from FreeBSD comes from the open source community. The bge(4) driver in HEAD supports the BCM5720 but does not work correctly on the Dell R720 systems due to support for iDRAC on the Dell systems. Driver changes have been submitted for HEAD for the Dell R720 but have not been added to head or back ported to 9.0. Hopefully the changes to HEAD will be available in the next few days.
> 
> Dave



Can the maintainer of this driver comment on the status of the integration of the R720 support for iDRAC?

Availability within the next few days would be really sweet.


----------



## yongari@ (May 7, 2012)

I'm working on integrating the required change to make BCM5720 with NC-SI firmware work on FreeBSD.  It needs more testing and cleanups to not introduce regressions on other controllers. I'll let you know when patch is available to testing.


----------



## chodong (May 7, 2012)

Thank you very much for the status update. I'll monitor this thread for further updates.


----------



## linuxunix (May 8, 2012)

Yongari,

I request you to help us with the driver and speed up your testing as I have no other option than Broadcom NIC. Hope you will understand and thanks a lot for the heads-up!

Sorry if I am just being curious, can I download the patch and test it from my end too?


----------



## linuxunix (May 17, 2012)

Yongari,

Any updates? We really need your quick help for this.
Please update.


----------



## yongari@ (May 18, 2012)

I've encountered a couple of issues during testing. Had been busy for $DAYJOB and couldn't find spare time to address it for a while. You can download experimental code if you can't wait for stable driver. The experimental driver may or may not work on your box.
http://people.freebsd.org/~yongari/bge/if_bge.c
http://people.freebsd.org/~yongari/bge/if_bgereg.h
Note, you need latest stable/9 to build the experimental driver.


----------



## linuxunix (May 18, 2012)

Thanks for sharing the experimental driver even when you are busy with your job.
May I know the expected time you gonna finish up with stable driver?

I always enjoy playing around FreeBSD but unable to get my NIC work and this has really made me stuck.
Please help me with the fix soon.I have been watching this space everyday since a week.


----------



## yongari@ (May 21, 2012)

I uploaded updated driver. It still has some issues but appear to work better than previous one.
http://people.freebsd.org/~yongari/bge/if_bge.c
http://people.freebsd.org/~yongari/bge/if_bgereg.h
http://people.freebsd.org/~yongari/bge/brgphy.c

You need latest stable/9 to try this driver.

Copy downloaded if_bge.c and if_bgereg.h to /usr/src/sys/dev/bge directory.
Copy downloaded brgphy.c to /usr/src/sys/dev/mii directory.
Rebuild kernel and reboot to take changes effect.


----------



## linuxunix (May 21, 2012)

Thanks Yongari...I will test it.


----------



## linuxunix (May 21, 2012)

Yongari,

I am using the RELEASE version of FreeBSD (I wonder if that will matter?).

```
freebsd9_0# uname -arn
FreeBSD freebsd9_0 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan  3 07:46:30 UTC 2012     
root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
```
Where can I download stable version?

I tried replacing if_bge.c and if_bgereg.h to the /usr/src/sys/dev/bge directory. Also, I replaced the downloaded brgphy under the mii directory with this new one. When I tried running:


```
#cd /usr/src
#make buildkernel KERNCONF=GENERIC
```

It threw these errors:


```
/usr/src/sys/dev/mii/brgphy.c:147: error: 'MII_MODEL_BROADCOM3_BCM5720C' undeclared here (not in a function)
/usr/src/sys/dev/mii/brgphy.c:147: error: 'MII_STR_BROADCOM3_BCM5720C' undeclared here (not in a function)
/usr/src/sys/dev/mii/brgphy.c:149: error: 'MII_MODEL_BROADCOM3_BCM57780' undeclared here (not in a function)
/usr/src/sys/dev/mii/brgphy.c:149: error: 'MII_STR_BROADCOM3_BCM57780' undeclared here (not in a function)
*** Error code 1

Stop in /usr/obj/usr/src/sys/GENERIC.
*** Error code 1

Stop in /usr/src.
*** Error code 1

Stop in /usr/src.
```

Any idea what could be the issue?


----------



## yongari@ (May 21, 2012)

You're using 9.0-RELEASE. As I said, you need latest 9-stable or CURRENT to build the driver. There were lots of mii(4) changes.


----------



## linuxunix (May 21, 2012)

Yongari,

Oh..I was little late to respond to you. I have understood that I need to make it to RELENG_9 (STABLE) through my CVSUP. I have started with the upgrading. Will let you know the result.


----------



## FreeBee (Jun 12, 2012)

Hi Yongari,

My company just purchased some Dell R720's with built-in BCM5720 NICs. Do you have any update on the development of the drivers? 

Thanks!


----------



## linuxunix (Jun 13, 2012)

*Driver is working !!!*

I created a CVSRepo for RELENG_9 and included your driver. It*'*s working like a charm.
Thanks a lot.


----------



## DrBaud (Jun 25, 2012)

Does RELENG_9 contain any updates from the files uploaded by yongari? If I do anything other than a ping bge reports a 
	
	



```
watchdog timeout -- resetting
```
 error.


----------



## chodong (Jul 9, 2012)

The new driver does not work.

It is only successful at ping 90% of the time and then the error "network is unreachable" occurs.

Please provide an update and an ETA for the complete driver.


----------



## Terry_Kennedy (Jul 9, 2012)

chodong said:
			
		

> The new driver does not work.
> 
> It is only successful at ping 90% of the time and then the error "network is unreachable" occurs.
> 
> Please provide an update and an ETA for the complete driver.


I realize this is causing problems for you. However, as far as I know, yongari@ is a volunteer (as are most FreeBSD developers) and not a Broadcom or Dell employee. I've found him to be quite responsive to problem reports. Of course, the more information that's provided, the easier it is to track down the underlying problem.

As a user-to-user suggestion, you might want to re-word your message to something like "I'm still having issues with the BCM5720 card. I'm seeing about a 10% packet loss rate, accompanied by 'network is unreachable' messages. What information should I collect to help you troubleshoot this issue?"

Some things I might try while waiting for a response are:


Is there a pattern to the 10% loss?
Does varying the packet size change the loss rate?
Do ierrs or oerrs show up in `# netstat -i`If so, one per dropped packet, or more?
Does the BCM5720 work successfully under some other operating system? (Boot from a LiveCD or the Broadcom diagnostic to rule out defective hardware.)
If it is critical to get this system into production and you have an open slot, you might want to consider adding a network card with known driver support (I'm using both Intel em(4) and Broadcom bce(4)/bge(4) cards with success). If the BCM5720 you're having the issue with is on an add-in card, you might just try swapping the card. But I'm guessing it is built onto the R720 mainboard.


----------



## chodong (Jul 9, 2012)

Terry,

Thank you for your input. I understand the situation quite well.

That the card won't ping is not a complex bug that is difficult to track. We are talking basics here.

I appreciate very much Yongari's volunteering of his time and if I were a network driver developer I would jump in
to help.

I hope there are others that might do so as the R720 is a popular platform and the BCM720 is the default LOM NIC.

I am currently running on dual-port NICs but this configuration cannot be delivered as a beta platform.

Please appreciate that I have project commitments to meet as well and that the cost of purchasing a dozen enterprise class quad-port NICs is not insignificant.

Having committed to FreeBSD I am in a difficult situation when a NIC driver is blocking my progress and the customer delivery looms ever closer.


----------



## yongari@ (Aug 8, 2012)

Sorry for late reply. I didn't forget this issue and I didn't give up either.
The experimental driver posted in above URL shows mixed results. It does not seem to work on Dell R720/R620 but I was told the driver works on Dell R420/R320(newer Dell model).
It seems there is some hardware differences on R720/R620 so it would be more helpful to know what is happening on these boxes.  If one of R720/R620 owners can setup a remote debugging environments for me it would help a lot to find root cause.
See the following URL for possible remote debugging setup. If you have resources for remote debugging, please send private mail to yongari@FreeBSD.org
http://people.freebsd.org/~yongari/remote_debugging.txt


----------



## linuxunix (Aug 12, 2012)

I built a new FreeBSD 9.0 with the experimental driver submitted by you. I installed on R720 and things are working fine for me. I have not testing it yet.

I have those servers(R720/R620) in labs but it is not open to outside world.
We need some volunteer to provide access to Yongari so that he can perform the testing.

I am keen on seeing matured product in this aspect, rather than experimental.


----------



## yongari@ (Sep 7, 2012)

I've updated WIP driver again(URL is the same). With the generous help from Emulab, I was able to narrow down root cause of the issue on remote server and finally fixed the instability issue on Dell R820. So it's highly recommended to try WIP driver if bge(4) still does not work on your box.


----------



## yongari@ (Oct 12, 2012)

I checked in the WIP version to CURRENT. bge(4) in CURRENT should support all Dell Rx20 and HP systems. I'll merge the change to stable/9 and stable/8 after 3 ~ 4 weeks.


----------



## omerfsen (Nov 2, 2012)

*9.1-RC2 or 9.1?*

Hi,

Does your commit will be in 9.1 ?

Regards.


----------



## omerfsen (Nov 5, 2012)

For anyone interested and for the sake of clarity i have just received an email from yongari@ that it was committed to HEAD and will not make it to FreeBSD 9.1 (but will appear on 9-Stable)

==============================================

No, it was just committed to HEAD. I'll merge the change to both
stable/9 and stable/8 after settlement but it wouldn't make it in
9.1(it's too late). Probably you can update to stable/9 to use the
controller after the merge.


----------



## omerfsen (Dec 17, 2012)

*any update*

Any update on this? Does this code commited to 9-Stable?

Regards.


----------



## yongari@ (Dec 18, 2012)

Yes, the change was merged to both stable/9 and stable/8 at Nov 26, 2012.


----------



## ads (Feb 18, 2013)

I am currently going through some testing of a Dell R420 server with a BCM 5720. We have FreeBSD 9.1-RELEASE installed on the system and we are getting:


```
bge0: link state changed to UP
bge0: link state changed to DOWN
```

about 1000 times during boot and periodically throughout the day.

We have tried a few suggestions to resolve this issue, like setting:


```
hw.bge.allow_asf="0"
hw.pci.enable_msi="0"
hw.pci.enable_msix="0"
```

in the loader.conf but we are still having no luck.

Any ideas?


----------



## Terry_Kennedy (Feb 20, 2013)

ads said:
			
		

> Any ideas?


Not an idea, but possibly pointing in the right direction: Does this server have an iDRAC that shares the Ethernet connector with the normal operating system network functions? Those are often implemented with oddball methods which the FreeBSD driver(s) may not handle. If the iDRAC thinks it needs to reset the network interface, it will do that regardless of the effect it has on the running system.

Many Dell PowerEdge systems have a pair of network ports on the mainboard. If your system has an em1 device, can you move the network cable (and your rc.conf settings) over to it and see if the problem persists? That would help pin it down to either a problem with the integrated management or a problem that affects all ports on that system.


----------



## ads (Feb 27, 2013)

The server doesn't have a em1 device.

I have decided to use the Intel I350 instead and this is fully supported.


----------



## sharkys (May 23, 2013)

Would be great to have some update regarding this as we are considering to switch on PowerEdge T320, mainly if in case of sharing network interface with iDRAC is causing issues or not and mainly if switching to secondary NIC (eg. not used in parallel as loadbalancer) helps (or even dedicated in case of iDRAC Enterprise). If I understood correctly, in case of iDRAC Express licence you can't choose dedicated NIC even if installed. Thank you.


----------



## Terry_Kennedy (May 23, 2013)

sharkys said:
			
		

> Would be great to have some update regarding this as we are considering to switch on PowerEdge T320, mainly if in case of sharing network interface with iDRAC is causing issues or not and mainly if switching to secondary NIC (eg. not used in parallel as loadbalancer) helps (or even dedicated in case of iDRAC Enterprise).


Unfortunately, the prior poster hasn't visited here since their post on 27 February. So we don't know if it was a hardware problem on their specific system or a general problem. On the other hand, you can see from this thread that @yongari@ can fix issues if there is access to a system for testing.



> If I understood correctly, in case of iDRAC Express licence you can't choose dedicated NIC even if installed. Thank you.


I believe you are correct.


----------



## ads (May 24, 2013)

sharkys said:
			
		

> Would be great to have some update regarding this as we are considering to switch on PowerEdge T320, mainly if in case of sharing network interface with iDRAC is causing issues or not and mainly if switching to secondary NIC (eg. not used in parallel as loadbalancer) helps (or even dedicated in case of iDRAC Enterprise). If I understood correctly, in case of iDRAC Express licence you can't choose dedicated NIC even if installed. Thank you.



Hi @sharky,

We unfortunately could not resolve this issue. We *i*nstead installed the Intel i350 onto the machines to get the networking running as Intel support FreeBSD officially*.*


----------



## sharkys (May 27, 2013)

I was trying to contact @yongari@ to eventually give him access for testing when the server will be available (and if the issue will appear). However, according to the logs, he hasn't logged in since 12/2012 ;-(


----------



## kpa (May 27, 2013)

This is a user to user forum for most part. The correct way to contact the developers is to post on a mailing list, freebsd-net in this case. You can try to send email directly to @yongari@, the email address should be yongari@freebsd.org. Posting to the mailing list first is preferred though, you could draw the attention of another developer who can also help you if @yongari@ is too busy.


----------



## sharkys (Jun 4, 2013)

So it's really strange. Despite of running iDRAC on dedicated port on DELL PowerEgde T320, seems that connected interface is "fluctuating" stating following right after boot hundreds of time:


```
bge0: link state changed to UP
bge0: link state changed to DOWN
```

...but not always. It occurred now after running system two days, after lockup (yes...) and reboot it starts flooding console and I wasn't able to connect, after another restart it is now running fine.

Additional interesting observation - fluctuation occurred after hard reset, but didn't occur after soft reset.

Also I added following in to the loader.conf as stated by @@ads and this seems to be working for me - so far no fluctuation. 


```
hw.bge.allow_asf="0"
hw.pci.enable_msi="0"
hw.pci.enable_msix="0"
```


----------

