# External IP on lo0 (Why?)



## surv (Jul 26, 2016)

```
# uname -rs
FreeBSD 11.0-BETA2
```
After boot:

```
# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS       wlan0
127.0.0.1          link#1             UH          lo0
192.168.1.0/24     link#2             U         wlan0
[COLOR=#ff0000]192.168.1.3        link#2             UHS         lo0[/COLOR]
```
If I add the line
`010 allow log all from any to any via lo0`
to ipfw config, in the logs I see it:
`kernel: ipfw: 10 Accept TCP 201.51.27.28:43 192.168.1.3:60689 [COLOR=#ff0000]in[/COLOR] via [COLOR=#ff0000]lo0[/COLOR]`
and similar connections from Internet IP's

This is normal ?


----------



## SirDice (Jul 26, 2016)

Yes. It's its own IP address. And there's only one destination to route it to; localhost.


----------



## Murph (Jul 26, 2016)

It's normal on recent versions (same on 10.3) to see it in `netstat -rn` for the UHS routes.  I believe the routing is setup like that to enable super-jumbo (yeah, I just made that term up) frames for local traffic which happens to use one of the external addresses, i.e. faster sockets / less overhead.  lo0 has a 16384 MTU, so you get much higher performance.  I'd need to dig deep into the code to be absolutely certain, but I believe it's just FreeBSD's way of essentially short-circuiting local traffic.  I say "recent versions" as I'm not about to go back and dig through history; it probably has been like that for a long time.

I do not believe that it is generally good practice to attach IPFW rules involving external addresses to lo0 (they should be attached to the appropriate external interface).  For best performance, lo0 should generally be minimised in firewall filtering or completely exempted from firewall processing (i.e. `setup_loopback()` from /etc/rc.firewall for IPFW, or `set skip on lo0` for PF).  There may be some quite specific circumstances where that is good and necessary (e.g. possibly some styles of jail / VM configs on a loopback network behind in-host NAT), just not something in a normal general/simple config.


----------



## surv (Jul 26, 2016)

SirDice said:


> Yes. It's its own IP address. And there's only one destination to route it to; localhost.


I thought that lo0 is intended only for "internal" use (me to me)



Murph said:


> I believe the routing is setup like that to enable super-jumbo (yeah, I just made that term up) frames for local traffic which happens to use one of the external addresses, i.e. faster sockets / less overhead.  lo0 has a 16384 MTU, so you get much higher performance.


Perhaps this explains why almost all connections from Internet go to wlan0 and only some go to lo0


			
				Murph said:
			
		

> I do not believe that it is generally good practice to attach IPFW rules involving external addresses to lo0 (they should be attached to the appropriate external interface).


Yes, but I added this rule just for logging of this connections


----------



## Mikalai (Aug 11, 2017)

If, following handbook, you setup two jails on external ips an some external interface, jails will be able to talk to each other, because of `set skip on lo0`, even when everything else is blocked.

I have tried to come up with a set of rules that may allow jail1 to serve jail2 on one port with everything else blocked. I seem to either block everthing, or allow everything via `set skip on lo0`.

Are there reliable pf rules for such common scenario?


----------



## Deleted member 30996 (Aug 11, 2017)

Murph said:


> I do not believe that it is generally good practice to attach IPFW rules involving external addresses to lo0 (they should be attached to the appropriate external interface).  For best performance, lo0 should generally be minimised in firewall filtering or completely exempted from firewall processing (i.e. `setup_loopback()` from /etc/rc.firewall for IPFW, or `set skip on lo0` for PF).  There may be some quite specific circumstances where that is good and necessary (e.g. possibly some styles of jail / VM configs on a loopback network behind in-host NAT), just not something in a normal general/simple config.



pf rule:

```
antispoof for lo0
```


----------



## Mikalai (Aug 11, 2017)

Trihexagonal said:


> pf rule:
> 
> ```
> antispoof for lo0
> ```


That is a host's IP on lo0, hence legitimate. May be antispoof can close it. Then connectivity is lost. Seems to be a good reason for telling everyone `set skip on lo0`.
Unfortunately this automagic host ip assign doesn't boil well with fortifying second perimeter in jails defense, i.e. when one of apps gets busted. Thread on related topic http://freebsd.1045724.x6.nabble.com/Firewalling-jails-and-lo0-td6120599.html Note how colorful is tcpdump.


----------



## Mikalai (Aug 11, 2017)

SirDice said:


> Yes. It's its own IP address. And there's only one destination to route it to; localhost.


Can you explain why this always happens, or point to any useful explanation? This seems to be a glaring hole in understanding of how networking is handled.
Can we even have a special doc page in a handbook for lo0? It touches the second perimeter in jails' defense.


----------



## Mikalai (Aug 11, 2017)

Murph said:


> It's normal on recent versions (same on 10.3) to see it in `netstat -rn` for the UHS routes.  I believe the routing is setup like that to enable super-jumbo (yeah, I just made that term up) frames for local traffic which happens to use one of the external addresses, i.e. faster sockets / less overhead.  lo0 has a 16384 MTU, so you get much higher performance.  I'd need to dig deep into the code to be absolutely certain, but I believe it's just FreeBSD's way of essentially short-circuiting local traffic.  I say "recent versions" as I'm not about to go back and dig through history; it probably has been like that for a long time.
> 
> I do not believe that it is generally good practice to attach IPFW rules involving external addresses to lo0 (they should be attached to the appropriate external interface).  For best performance, lo0 should generally be minimised in firewall filtering or completely exempted from firewall processing (i.e. `setup_loopback()` from /etc/rc.firewall for IPFW, or `set skip on lo0` for PF).  There may be some quite specific circumstances where that is good and necessary (e.g. possibly some styles of jail / VM configs on a loopback network behind in-host NAT), just not something in a normal general/simple config.



I want to express that we shouldn't down-play this, by saying that it is "not something normal".
Anyone who had more than one jail on a host, has weakened network filtering defense.


----------



## SirDice (Aug 11, 2017)

Mikalai said:


> I want to express that we shouldn't down-play this, by saying that it is "not something normal".
> Anyone who had more than one jail on a host, has weakened network filtering defense.


Jails provide process separation/isolation, not network separation.


----------



## Mikalai (Aug 11, 2017)

SirDice said:


> Jails provide process separation/isolation, not network separation.


That what it smells like. At the time of jail creation, people were not planning on putting 20 jails that have parts from whole web stack, including horizontal splitting of mono-apps into micro services (it reduces blast radius on intrusion).
Localhost, I guess, is by some RFC treats all local traffic equally trusting.

I fill a little betrayed. There is zero warning signs about addresses being put to lo0. And there is tons of advertisement of jails' strong isolation. Except today, word isolation is also loaded with network separation, cause all processes are networked. In fact isolation, isolation with no leaks.
On another hand, it only has been in a last couple of years that we have more secure ipc flags and implementation for running things like Postgres.


----------



## Mikalai (Aug 11, 2017)

SirDice said:


> Jails provide process separation/isolation, not network separation.


Then separation is a leaky concept, by virtue of being non-trivial.
I don't care if intrusion happens via unix or network sockets attack surface.


----------



## Mikalai (Aug 11, 2017)

SirDice said:


> Jails provide process separation/isolation, not network separation.


Yep! Unix way! Then we use pf in trying to achieve network separation. Unix way!
But, lo0's magic lets us down, playing against filtering. Is it time to call it a bug?


----------

