# Gateway problem



## phijazz (Sep 27, 2022)

Hello there, 
I have a small problem. I am trying to make a gateway with pf and dhcpd, on a lenovo m93p tiny, on which I added an ethernet interface via USB. 
(https://www.amazon.fr/gp/product/B085DJLDBC/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1)

So far, I have been able to link everything, establish connections on external and internal devices, make the gateway and ping external IP and url, make updates and download large files from the internal network, but when I surf from my laptop, the web pages can't load (I think it concerns only dynamic pages)
My modem is of course in bridge mode.

In pf.conf, the nat is set, the scrub too, I don't understand the problem...


----------



## SirDice (Sep 27, 2022)

Did you enable routing? `sysrc gateway_enable="YES"`


----------



## phijazz (Sep 27, 2022)

Yes. Otherwise, I couldn't ping any external IP or URL from my internal network


----------



## SirDice (Sep 27, 2022)

phijazz said:


> the web pages can't load (I think it concerns only dynamic pages)


A webpage is a webpage, doesn't matter if it's dynamic or not. That's totally irrelevant for the HTTP protocol.

What does your pf.conf look like?


----------



## phijazz (Sep 27, 2022)

I don't undestand why, but certain elements can't load while other can.

Here is my pf.conf

```
################ Macros ######################################################

### Interfaces ###
 ext_if ="em0"
 int_if ="ue0"
 all_if ="{ em0, ue0 }"

### ports ###
 icmp_types="{ echoreq unreach }"

### Queues, States and Types ###
 IcmpPing ="icmp-type 8 code 0"
 SshQueue ="(ssh_bulk, ssh_login)"
 SynState ="flags S/UAPRSF synproxy state"
 TcpState ="flags S/UAPRSF modulate state"
 UdpState ="keep state"

### Stateful Tracking Options (STO) ###
 OpenSTO ="(max 90000, source-track rule, max-src-conn 1000, max-src-nodes 256)"
 SmtpSTO ="(max   200, source-track rule, max-src-conn   10, max-src-nodes 256, max-src-conn-rate 200/30)"
 SshSTO  ="(max   100, source-track rule, max-src-conn   10, max-src-nodes 100, max-src-conn-rate 100/30,  overload <BLOCKTEMP> flush global)"
 WebSTO  ="(max  8192, source-track rule, max-src-conn  128, max-src-nodes 512, max-src-conn-rate 500/100)"

### Tables ###
 table <BLOCKTEMP> counters
 table <BLOCKPERM> counters file "/var/cache/pfblock"
 table <jails> persist


################ Options #####################################################

#############################
################ Misc Options

 set skip on lo
 set debug urgent
 set block-policy drop
 set loginterface $ext_if
 set state-policy if-bound
 set fingerprints "/etc/pf.os"
 set ruleset-optimization none
 
#############################
############# Timeout Options

 set optimization normal
 set timeout { tcp.closing 60, tcp.established 7200}

################ Normalization ###############################################

 #set-tos 0x2E is Maximize-Reliability + Minimize-Delay + Maximize-Throughput
 scrub in on $all_if all fragment reassemble

################ Translation #################################################

### NAT and Redirection rules

#############################
######## bastille and $int_if

 #nat on $ext_if from <jails> to any -> ($ext_if)
 rdr-anchor "rdr/*"
 nat on $ext_if inet from !($ext_if) to any -> ($ext_if)

#Matrix
 rdr on $ext_if inet proto tcp from  !($ext_if) to ($ext_if) port  8008 -> 10.0.2.2

# DENY rogue redirection
 no rdr


################ Filtering ###################################################

#############################
################ WAN Filtrage

#############################
###################exclusions

 block drop in  log quick on $ext_if           from <BLOCKPERM> to any
 block drop in  log quick on $ext_if proto udp from <BLOCKTEMP> to any

#############################
####### $ext_if default block

 antispoof for $all_if
 block drop in log on $ext_if

#############################
############# $ext_if inbound

#http https
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if) port https $TcpState $WebSTO
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if) port www   $TcpState $WebSTO

#mail
 pass in log on $ext_if inet proto tcp  from !($ext_if)        to ($ext_if) port { 25, 465, 587, 2525 }  $TcpState $OpenSTO
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if) port { 143, 993 }  $TcpState $OpenSTO

#ssh
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if) port 31336 $TcpState $OpenSTO

#mpd
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if) port 6600  $TcpState $OpenSTO

# Samba
 pass in log on $ext_if inet proto udp  from !($ext_if)        to ($ext_if) port { 137, 138 } $UdpState $OpenSTO
 pass in log on $ext_if inet proto tcp  from !($ext_if)     to ($ext_if) port { 139, 445 } $TcpState $OpenSTO
 
# Nfs
 pass in log on $ext_if inet proto { tcp, udp }  from !($ext_if)      to ($ext_if) port { 111, 2049, 20048 } $TcpState $OpenSTO

# Matrix sydent
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if) port 8448  $TcpState $OpenSTO
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to 10.0.2.2 port 8008  $TcpState $OpenSTO

# Spotifyd
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if) port {5353, 4444} $TcpState $OpenSTO
 pass in log on $ext_if inet proto udp  from !($ext_if)      to ($ext_if) port 5353 $UdpState $OpenSTO

# Torrent
 pass in log on $ext_if inet proto udp  from !($ext_if)      to ($ext_if) port {6771, 51413} $UdpState $OpenSTO
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if) port 51413 $TcpState $OpenSTO

# VNC
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if)  port 5900 $TcpState $OpenSTO

# ping
 pass quick inet proto icmp icmp-type $icmp_types
 pass quick inet6 proto icmp6 icmp6-type $icmp6_types

#############################
############ pass out network

 pass out on $ext_if proto tcp all modulate state flags S/SA
 pass out on $ext_if proto { udp, icmp } all keep state

#############################
####### pass internal network

 pass in on $int_if from $int_if:network to any keep state
 pass out on $int_if from any to $int_if:network keep state
```


----------



## SirDice (Sep 27, 2022)

```
#http https
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if) port https $TcpState $WebSTO
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if) port www   $TcpState $WebSTO
```
Do you have a web server running? These are for incoming connections from the internet. 


```
#mail
 pass in log on $ext_if inet proto tcp  from !($ext_if)        to ($ext_if) port { 25, 465, 587, 2525 }  $TcpState $OpenSTO
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if) port { 143, 993 }  $TcpState $OpenSTO
```
Do you have _incoming_ email? If not, remove these.


```
# Samba
 pass in log on $ext_if inet proto udp  from !($ext_if)        to ($ext_if) port { 137, 138 } $UdpState $OpenSTO
 pass in log on $ext_if inet proto tcp  from !($ext_if)     to ($ext_if) port { 139, 445 } $TcpState $OpenSTO
```
Yeah, don't do this. Do NOT allow unfettered access to your windows shares from the internet. That's a bad idea.


```
# Nfs
 pass in log on $ext_if inet proto { tcp, udp }  from !($ext_if)      to ($ext_if) port { 111, 2049, 20048 } $TcpState $OpenSTO
```
Same for NFS, why are you allowing random access from the internet to your NFS? Bad idea.


```
# VNC
 pass in log on $ext_if inet proto tcp  from !($ext_if)      to ($ext_if)  port 5900 $TcpState $OpenSTO
```
Bad idea to allow VNC access from a random internet address. 

Going through this list, I get the feeling you're not quite understanding the difference between _incoming_ and _outgoing_ connections.

Moving this thread to "Firewalls", as it has more to do with PF than networking.


----------



## phijazz (Sep 27, 2022)

1/ I have nginx and opensmtpd

2/ Its voluntary and I know there are some risks. 
Thank you for this advice, but it has nothing to do with my problem.
Do you see something who could explain my problem?


----------



## SirDice (Sep 27, 2022)

phijazz said:


> Its voluntary and I know there are some risks.


Understatement of the year in regard to SMB, NFS and VNC.



phijazz said:


> Do you see something who could explain my problem?




```
pass out on $ext_if proto tcp all modulate state flags S/SA
```
Change this to

```
pass out on $ext_if proto tcp all
```
modulate state has some issues in some cases. And `flags S/SA` could be problematic too.


----------



## phijazz (Sep 27, 2022)

SirDice said:


> Understatement of the year in regard to SMB, NFS and VNC.


Ok


SirDice said:


> Change this to
> 
> ```
> pass out on $ext_if proto tcp all
> ...


I did it, but the problem is still there.
I also tried previously with the following rule without it working:

```
pass out keep state
```


----------



## SirDice (Sep 27, 2022)

What exactly is the issue you're having on the laptop? Any error messages? Have you taken a look with tcpdump(1) to see what's going on? Check the traffic on the internal interface, and compare with the outgoing interface. Ideally you should only see NAT changing the source address and port of the connection.


----------



## diizzy (Sep 27, 2022)

I wonder if it has do with the all the filtering of TCP states and whatnot.

To debug I'd suggest that you comment out the following at least temporary


```
set state-policy if-bound
scrub in on $all_if all fragment reassemble
```

What does `sysctl net.inet.ip.forwarding` return on the router/firewall?

I would also highly recommend that you do follow SirDice's advice regarding what to allow or not.


----------



## covacat (Sep 27, 2022)

i had a similar symptom (shitty web experience) when using pppoe (mpd5)
fix was mssfix but in my case devices on the lan had a larger MTU than the WAN link


----------



## SirDice (Sep 27, 2022)

If his modem (in bridge mode) is anything like mine, you will just get a regular 1500 MTU from it. And you just set your external interface on the FreeBSD host to DHCP to get your internet address directly. It shouldn't cause any MTU issues. But it's always good to double check of course.


----------



## phijazz (Sep 28, 2022)

Thanks for your answers.

I will look with tcpdump.
pftop gives me a lot of TIME_WAIT:TIME_WAIT and FIN_WAIT_2:FIN_WAIT_2

On my side, I tried with a fresh BSD installation on a USB key to see if it could work.
I have the same problems, and this with other more standard pf.conf models.
I noticed however that if I put my modem back as a router (so with double-nat...), the BSD gateway works. But this is clearly not the goal. 

Is it possible that this is due to a protocol problem with my ISP?
I noticed that in bridge mode, I only have an inet6 address in fe80:...


----------



## SirDice (Sep 28, 2022)

phijazz said:


> I noticed that in bridge mode, I only have an inet6 address in fe80:...


Same with my provider. If I set the modem in 'router' mode, I get IPv6 addresses. When the modem is in bridge mode I don't get an IPv6 address at all. Provider only supports IPv6 if the modem is in router mode. I can highly recommend getting a free IPv6 tunnelbroker from HE (https://tunnelbroker.net/) if you really want to have IPv6. It's been running for at least a decade now, rarely had any issues with it. The only downside of using an IPv6 tunnelbroker is that Netflix, Disney+ and a couple of others claim I'm using a proxy and block access. So I have set my NVidia Shield (I use it mainly for viewing Netflix, Disney+, etc) to disable IPv6.


----------



## phijazz (Sep 28, 2022)

covacat said:


> i had a similar symptom (shitty web experience) when using pppoe (mpd5)
> fix was mssfix but in my case devices on the lan had a larger MTU than the WAN link


This!

Solved with this solution!!








						Solved - My ISP giving a MTU of 576
					

Hello all, I have a little server with one nic on WAN and the other LAN, my WAN is set on DHCP cause i dont have a static IP. Now my ISP (cable) is passing the info but it set the MTU to 576 which give me all sort of problem so i try to force it to 1500 using ifconfig re1 mtu 1500. which work...




					forums.freebsd.org


----------

