# FreeBSD 8 / ALTQ / em0 - reality check (confirmed bug - test)



## DutchDaemon (Aug 30, 2009)

If someone is using the combination mentioned in the topic subject: can you confirm that queueing currently does *not* work on em(4)? Specifically: *ALTQ-CBQ*.

I had the same altq ruleset running just fine on FreeBSD 7.2 with an fxp(4) interface, and I just tested an additional altq ruleset on this box's other interface (re(4)). Queueing works just fine there.

The altq ruleset has been tried and tested over the years, and it is just fine. I just changed the interface name from fxp0 to em0 for this new machine. 

For some reason, when using altq on em0 the queues show up fine in [cmd=]pfstat -sq -vv[/cmd], in [cmd=]pftop[/cmd] (option 8), and in pfstat's graphs, but no traffic ends up in them (not even in the root_em0 queue) -- well, maybe 20 packets in total today (there've been several GBs of traffic crossing em0).

If needs be, I will switch em0 and re0 to get queueing working again, but I hope someone is running the same setup. Let me know what you see.


----------



## aragon (Aug 30, 2009)

According to altq(4) it should work with em(4).  Possible regression if not...


----------



## DutchDaemon (Aug 30, 2009)

Yes, I saw 'em' listed there, so I went ahead and used it without too much thought. And then my stats remained blissfully empty. I switched to the other NIC on that server for now (re0), and it queues fine. So it's all down to em0 here. I'll see about a PR for this.

For the record:

```
em0: <Intel(R) PRO/1000 Network Connection 6.9.14>

em0@pci0:4:0:0: class=0x020000 card=0x01d11028 chip=0x109a8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Intel PRO/1000 PL Network Adaptor (82573L)'
    class      = network
    subclass   = ethernet
```
32-bits FreeBSD 8.0-BETA3 (csupped to today)

```
/*$FreeBSD: src/sys/dev/e1000/if_em.c,v 1.21.2.2 [B]2009/08/19[/B] 18:08:50 delphij Exp $*/
```
Pretty recent change, it seems.


----------



## DutchDaemon (Aug 31, 2009)

The change mentioned above appears to be this one:
http://lists.freebsd.org/pipermail/freebsd-net/2009-August/022758.html

I wonder if this could have any regression effects on altq functionality on em?

P.S.: this is the first time I use altq on em, so I have no idea whether it worked on FreeBSD 7.x, or FreeBSD 8.x prior to the installation date (Aug 30).


----------



## aragon (Aug 31, 2009)

That change seems rather minor.  I think it goes back farther to the big igb/em changes a while ago.  Try edit /usr/src/sys/dev/e1000/if_em.c:


```
--- if_em.c.orig	2009-08-31 19:16:50.000000000 +0200
+++ if_em.c	2009-08-31 19:14:46.000000000 +0200
@@ -62,6 +62,7 @@
 
 #include <net/bpf.h>
 #include <net/ethernet.h>
+#define ALTQ 1
 #include <net/if.h>
 #include <net/if_arp.h>
 #include <net/if_dl.h>
```

Then recompile that module.  If it fixes it, it might need a PR filed, or maybe ALTQ needs to be defined in src.conf now.


----------



## DutchDaemon (Aug 31, 2009)

There's already a PR filed at Intel for this, so for now I'll wait and see how that evolves. BTW, I have em and altq together in the kernel.


----------



## peachlover (Jan 5, 2010)

*PF: Altq not working on em (intel) interfaces*

Hello, ive been searching for solution of my problem along the forum and the only thing that is close to my case is explained in this thread by DutchDaemon

I'm using 8.0 i386 emulated on VirtualBox as guest , with rc.conf:

```
gateway_enable="YES"
hostname="peach.dcc.bg"
ifconfig_em0="DHCP"
hostname="peach.dcc.bg"
ifconfig_em1="inet 11.0.0.1 netmask 255.255.255.0"
pf_enable="YES"
pf_rules="/etc/pf.conf"
dhcpd_enable="YES"
dhcpd_iface="em1"
firewall_enable="YES"
firewall_type="open"
```

pf.conf:

```
ext_if = "em0"
internal_net = "11.0.0.0/24"
int_if = "em1"

scrub in all


altq on $ext_if cbq bandwidth 1Mb queue { default, http }
        queue default bandwidth 30% cbq(default)
        queue http bandwidth 70% cbq(borrow)


nat on $ext_if from $internal_net to any -> ($ext_if)


antispoof for $int_if

pass out on $ext_if all queue default
pass out on $ext_if proto tcp from $internal_net to any keep state queue http
```


and my kernel configuration (ill post only the end part of it):

```
# PCI Ethernet NICs.
device          em              # Intel PRO/1000 Gigabit Ethernet Family
device          miibus          # needed by pcn
device          pcn             # Amd Amnet Am79C973

# Pseudo devices.
device          loop            # Network loopback
device          random          # Entropy device
device          ether           # Ethernet support
device          tun             # Packet tunnel.
device          pty             # BSD-style compatibility pseudo ttys
device          md              # Memory "disks"
device          firmware        # firmware assist module

device          bpf             # Berkeley packet filter
device          pf
device          pflog
device          pfsync

options         ALTQ
options         ALTQ_CBQ
options         ALTQ_RED
options         ALTQ_RIO
options         ALTQ_HFSC
options         ALTQ_PRIQ
options         ALTQ_NOPCC
options         IPFIREWALL
options         DUMMYNET
options         HZ=1000
```

So just as DutchDaemon explained there is no packets passing the queues or maybe just a few per hour. The Nat is ok its working fine the only part thats not working is ALTQ. Its shown in pftop that the rule is triggered so it matches the traffic flow. But nothing goes to the altq em0_root or any other of my queues.

I've tested the speed with mikrotik btest utility and its around TX:3.0 Mb RX:20 Mb which is not the case for anything bigger than 1Mb because its set in the rule as bandwidth for the interface.

Is it the em driver. Is it some version issue or something coming from the emulation. My bsd interfaces are bridged to the physical.


Any help will be appreciated. Thanks in advance


----------



## DutchDaemon (Jan 5, 2010)

[ previous post merged in due to similarities ]


----------



## peachlover (Jan 17, 2010)

*Something ?*

Is there some news about this issue? 
Dutch, please share some more information on your configuration (is it some kind of virtual client, if is it so what is the VM host and with that VNET configuration). Or if you found a solution what it is (change of the IF is considered a solution too).
Thanks in advance! :i


----------



## DutchDaemon (Jan 17, 2010)

My situation is not resolved (but I haven't tested ALTQ on em0 since I switched interfaces, and I haven't tracked driver activity since), but it is a 'bare metal' situation for me. No virtualisation or anything, just a real interface on a real system.


----------



## adri (Jan 25, 2010)

I have the same problem with altq(hfsc)+em on a Dell 1850, using Release 8.0-p2.
No virtualisation and I need to use altq for voip, but same symptoms as DutchDaemon, only a few packets get redirected to the queue.
If I had an option to install a few extra nic cards, I would, but the 1850 only has 1 PCI expansion slot, and I need at least 2.

A PR was filed on August 2009, but not much has happened after that.
I hope the problem will be found soon, even if it means running 8-Stable instead of 8-Release.

Adri.


----------



## TinyCamel (Jan 27, 2010)

I have the same problem. I am using Release 8.0-p2, and I just found that altq on em0 really did nothing!!! What the hell! If I remembered correctly, em driver used to be working, but now it's dead! can anyone fix it? Thanks in advance!!


----------



## DutchDaemon (Jan 31, 2010)

There is some activity on the PR:



> We have been able to successfully replicate this problem on FreeBSD 8 RELEASE p2.  We have also create a custom kernel using the FreeBSD 8 sources but replacing sys/dev/e1000 with the files in CVS.  Using what is in head (as of 29-JAN-2010) results in the same behavior where altq is not functional.  Replacing the e1000 files with the version in CVS that is tagged as for FreeBSD 7.2 results in a kernel that does not exhibit the behavior.  Please let us know if you need any assistance with replicating this bug or testing a patch for it.  This problem is causing us significant problems and we would like to avoid deploying a Frankenstein FreeBSD 8 with FreeBSD 7.2 e1000 drivers if possible.  Thanks in advance.



http://www.freebsd.org/cgi/query-pr.cgi?pr=138392


----------



## peachlover (Feb 1, 2010)

So still no resolution of the problem outside the Frankenstein FreeBSD 8.0 with 7.2 em module?


----------



## DutchDaemon (Feb 1, 2010)

Jack Vogel (who works on the driver) told me that "Getting driver changes into 7.3 takes priority but this will get some time." So we'll have to be patient. BTW, I mistook the response on the PR (two posts back) as 'activity towards a driver', but it was in fact someone else who reported problems and offered some input/findings and assistance in testing patches.


----------



## DutchDaemon (Feb 1, 2010)

News from Jack:



> Are you guys set up to test this stuff? It has been suggested that the simple addition
> of a header file will do, and if its that simple I'd sure like to know
> 
> What you do is add  #include "opt_altq.h" at the top of if_em.c, right after
> "opt_inet.h", can you try that and let me know if it fixes things?



If some of you could test this, that would be nice. The file he's talking about is /usr/src/sys/dev/e1000/if_em.c, and the context is:


```
#ifdef HAVE_KERNEL_OPTION_HEADERS
#include "opt_device_polling.h"
#include "opt_inet.h"
#endif
```


----------



## simonlok (Feb 1, 2010)

I've received this patch:

http://stg2.net/tmp/fbsd8_kernel_em_altq_variants/em_igb_altq.patch


I've compiled three kernels in case you guys want to test this stuff out.

This one is a FreeBSD 8 kernel with the FreeBSD 8 release [em] driver that has been patched with the altq patch:

http://stg2.net/tmp/fbsd8_kernel_em_altq_variants/kernel_fbsd8_altq_patched.tgz

This one is a FreeBSD 8 kernel with the FreeBSD [em]  (src/sys/dev/e1000) driver upgraded to the latest in CVS head as of today (01-FEB-2010) and the driver patched with the altq patch.

http://stg2.net/tmp/fbsd8_kernel_em_altq_variants/kernel_fbsd8_e1000_head_altq_patched.tgz


This is a FreeBSD 8 kernel with the [em] driver (src/sys/dev/e1000) downgraded to the one that is included in FreeBSD 7.2 release.  This kernel is confirmed working on our deployments.

http://stg2.net/tmp/fbsd8_kernel_em_altq_variants/kernel_fbsd8_e1000_from_72.tgz


----------



## tanakorn (Feb 8, 2010)

simonlok said:
			
		

> I've received this patch:
> 
> http://stg2.net/tmp/fbsd8_kernel_em_altq_variants/em_igb_altq.patch



I tried this patch but it still doesn't work.


----------



## adri (Feb 12, 2010)

There is a patch on the freebsd-stable mailing list, which seems to work.
When I apply the patch, traffic gets queued on altq again.

Adri.


----------



## simonlok (Feb 14, 2010)

It appears that a fix has been applied to the FreeBSD tree.

http://svn.freebsd.org/viewvc/base?view=revision&revision=203834

However, the committer (Max Laier) has noted that there still remains a problem:



> Note that em, igb and ixgbe are still subtly broken after this.  The problem
> is that they all call drbr_enqueue after failure to transmit a previously
> dequeued packet.  This might mess up the packet order as the buf_rings are
> strictly FIFO.  The fix is to make sure that there are enough resources to
> ...



We have verified that the patch on which the commit is based on does indeed fix the problem.  We just aren't sure whether to continue using the FreeBSD 8 kernel with the 7.2 e1000 directory or moving to the 8.0 with the CVS head e1000 directory is the right way to proceed.

Any thoughts?


----------



## DutchDaemon (Feb 14, 2010)

I see that they plan to MFC after four weeks. I'll wait for that myself.


----------



## wonslung (Jul 19, 2010)

so is this fixed or not?  I wasn't away of this issue, i thought i had done something wrong.......

on my system, the queues show up but everything seems to hit only the BULK queue....


does this work in 8.1?


----------



## schmurfy (Sep 24, 2010)

I recently tried to use freebsd on a box with em drivers and the results I got are really odd.
I used a 8.0-stable freebsd tree and tried different combination for the sys/dev/e1000 subfolder, my test subject is a dedicated server hardware with four em network cards used to route data and voip traffic, the pf configuration is working on another device and I used exactly the same so I know the QoS configuration works.

8.0-stable: not working at all, packets sometimes appear on a pfctl -vvs queue but after a reload all counters stayed at 0
8.1-stable : somewhat working but some phone calls have sound glitch even with no traffic except the phone call
7.2-stable : same as 8.1
HEAD: kernel does not even boot, strange thing is that it does not detect the cpu class but I used the same config as the other tests, but hey it's HEAD after all xD

Is there any combination that is supposed to work or is the em driver completely broken with altq currently ?


----------

