# Outbound traffic problem



## spag (Sep 4, 2020)

So problem is that several FreeBSD *12.1-RELEASE-p9* machines with interfaces *em0*, *em1,* *tun0* from a while start giving an outbound connectivity errors.
It goes like when trying to connect external IP with SSH, FTP, NFS or HTTPS - 4 connections out of 10 are failed: *connection timeout.*
Once connection is established all is working fine.  For example `poundiere` is building 1000 ports on one of the servers and it is getting like 20 failed build due to a problem connecting with GitHub or others.

I am not quite sure where to look at it. I did try:

/var/log/messages looks clean and healthy.
/etc/pf.conf is set up and I thought it might be a problem there but config was unchanged for years now,
Only system updates via `freebsd-update fetch install`
`pflog0` does not show anything that gets blocked.
I did try net/mtr-nox11 to the various IP addresses and there is no packet drop. Interesting is sometimes `mtr`give me protocol error when adding -T and -u options. 2 in 10.
`traceroute` to a number of IP addresses works without any drops.
`telnet` to IP's on different ports is random: >4 on 10 fail to different IP's.
`nc` to IP's on different ports is random: >4 on 10 fail to different IP's.
`wget` from jails fail like 50% to the same IP ( not domain names ).
DNS is working fine.

I have a few theories:

a firewall above servers is just messing around
a router above cannot pass out packages
`pfctl` is doing too much out filtering ( nothing in logs or `pflog` ) - plan to disable it.
operating system get delays somewhere
a network interface is faulty ( but at the same time on multiple servers ? )
Any help on how would be much appreciated.


```
#netstat -s
tcp:
        15022 packets sent
                9415 data packets (1482010 bytes)
                1 data packet (768 bytes) retransmitted
                0 data packets unnecessarily retransmitted
                0 resends initiated by MTU discovery
                1454 ack-only packets (28 delayed)
                0 URG only packets
                0 window probe packets
                0 window update packets
                4236 control packets
        9907 packets received
                8052 acks (for 1482472 bytes)
                7 duplicate acks
                0 acks for unsent data
                2623 packets (229795 bytes) received in-sequence
                0 completely duplicate packets (0 bytes)
                0 old duplicate packets
                0 packets with some dup. data (0 bytes duped)
                0 out-of-order packets (0 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                0 window update packets
                2 packets received after close
                4 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
                0 discarded due to memory problems
        3715 connection requests
        7 connection accepts
        0 bad connection attempts
        0 listen queue overflows
        0 ignored RSTs in the windows
        526 connections established (including accepts)
                517 times used RTT from hostcache
                517 times used RTT variance from hostcache
                0 times used slow-start threshold from hostcache
        3679 connections closed (including 2 drops)
                10 connections updated cached RTT on close
                10 connections updated cached RTT variance on close
                0 connections updated cached ssthresh on close
        3030 embryonic connections dropped
        7351 segments updated rtt (of 10530 attempts)
        242 retransmit timeouts
                0 connections dropped by rexmit timeout
        0 persist timeouts
                0 connections dropped by persist timeout
        0 Connections (fin_wait_2) dropped because of timeout
        2 keepalive timeouts
                0 keepalive probes sent
                2 connections dropped by keepalive
        147 correct ACK header predictions
        1293 correct data packet header predictions
        8 syncache entries added
                0 retransmitted
                0 dupsyn
                0 dropped
                7 completed
                0 bucket overflow
                0 cache overflow
                1 reset
                0 stale
                0 aborted
                0 badack
                0 unreach
                0 zone failures
        8 cookies sent
        0 cookies received
        4 hostcache entries added
                0 bucket overflow
        0 SACK recovery episodes
        0 segment rexmits in SACK recovery episodes
        0 byte rexmits in SACK recovery episodes
        0 SACK options (SACK blocks) received
        0 SACK options (SACK blocks) sent
        0 SACK scoreboard overflow
        0 packets with ECN CE bit set
        0 packets with ECN ECT(0) bit set
        0 packets with ECN ECT(1) bit set
        0 successful ECN handshakes
        0 times ECN reduced the congestion window
        0 packets with matching signature received
        0 packets with bad signature received
        0 times failed to make signature due to no SA
        0 times unexpected signature received
        0 times no signature provided by segment
        0 Path MTU discovery black hole detection activations
        0 Path MTU discovery black hole detection min MSS activations
        0 Path MTU discovery black hole detection failures
TCP connection count by state:
        0 connections in CLOSED state
        27 connections in LISTEN state
        0 connections in SYN_SENT state
        0 connections in SYN_RCVD state
        5 connections in ESTABLISHED state
        0 connections in CLOSE_WAIT state
        0 connections in FIN_WAIT_1 state
        0 connections in CLOSING state
        0 connections in LAST_ACK state
        0 connections in FIN_WAIT_2 state
        57 connections in TIME_WAIT state
udp:
        1800 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        0 with no checksum
        61 dropped due to no socket
        6 broadcast/multicast datagrams undelivered
        0 dropped due to full socket buffers
        0 not for hashed pcb
        1733 delivered
        1862 datagrams output
        0 times multicast source filter matched
sctp:
        0 input packets
                0 datagrams
                0 packets that had data
                0 input SACK chunks
                0 input DATA chunks
                0 duplicate DATA chunks
                0 input HB chunks
                0 HB-ACK chunks
                0 input ECNE chunks
                0 input AUTH chunks
                0 chunks missing AUTH
                0 invalid HMAC ids received
                0 invalid secret ids received
                0 auth failed
                0 fast path receives all one chunk
                0 fast path multi-part data
        0 output packets
                0 output SACKs
                0 output DATA chunks
                0 retransmitted DATA chunks
                0 fast retransmitted DATA chunks
                0 FR's that happened more than once to same chunk
                0 output HB chunks
                0 output ECNE chunks
                0 output AUTH chunks
                0 ip_output error counter
        Packet drop statistics:
                0 from middle box
                0 from end host
                0 with data
                0 non-data, non-endhost
                0 non-endhost, bandwidth rep only
                0 not enough for chunk header
                0 not enough data to confirm
                0 where process_chunk_drop said break
                0 failed to find TSN
                0 attempt reverse TSN lookup
                0 e-host confirms zero-rwnd
                0 midbox confirms no space
                0 data did not match TSN
                0 TSN's marked for Fast Retran
        Timeouts:
                0 iterator timers fired
                0 T3 data time outs
                0 window probe (T3) timers fired
                0 INIT timers fired
                0 sack timers fired
                0 shutdown timers fired
                0 heartbeat timers fired
                0 a cookie timeout fired
                0 an endpoint changed its cookiesecret
                0 PMTU timers fired
                0 shutdown ack timers fired
                0 shutdown guard timers fired
                0 stream reset timers fired
                0 early FR timers fired
                0 an asconf timer fired
                0 auto close timer fired
                0 asoc free timers expired
                0 inp free timers expired
        0 packet shorter than header
        0 checksum error
        0 no endpoint for port
        0 bad v-tag
        0 bad SID
        0 no memory
        0 number of multiple FR in a RTT window
        0 RFC813 allowed sending
        0 RFC813 does not allow sending
        0 times max burst prohibited sending
        0 look ahead tells us no memory in interface
        0 numbers of window probes sent
        0 times an output error to clamp down on next user send
        0 times sctp_senderrors were caused from a user
        0 number of in data drops due to chunk limit reached
        0 number of in data drops due to rwnd limit reached
        0 times a ECN reduced the cwnd
        0 used express lookup via vtag
        0 collision in express lookup
        0 times the sender ran dry of user data on primary
        0 same for above
        0 sacks the slow way
        0 window update only sacks sent
        0 sends with sinfo_flags !=0
        0 unordered sends
        0 sends with EOF flag set
        0 sends with ABORT flag set
        0 times protocol drain called
        0 times we did a protocol drain
        0 times recv was called with peek
        0 cached chunks used
        0 cached stream oq's used
        0 unread messages abandonded by close
        0 send burst avoidance, already max burst inflight to net
        0 send cwnd full avoidance, already max burst inflight to net
        0 number of map array over-runs via fwd-tsn's
ip:
        21201 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with ip length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 packets reassembled ok
        21072 packets for this host
        0 packets for unknown/unsupported protocol
        0 packets forwarded (0 packets fast forwarded)
        0 packets not forwardable
        0 packets received for unknown multicast group
        0 redirects sent
        24519 packets sent from this host
        2224 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        4 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 tunneling packets that can't find gif
        0 datagrams with bad address in header
icmp:
        65 calls to icmp_error
        0 errors not generated in response to an icmp message
        Output histogram:
                echo reply: 5073
                destination unreachable: 65
        0 messages with bad code fields
        0 messages less than the minimum length
        0 messages with bad checksum
        0 messages with bad length
        0 multicast echo requests ignored
        0 multicast timestamp requests ignored
        Input histogram:
                echo reply: 117
                destination unreachable: 51
                echo: 5073
                time exceeded: 4136
        5073 message responses generated
        0 invalid return addresses
        0 no return routes
        ICMP address mask responses are disabled
igmp:
        0 messages received
        0 messages received with too few bytes
        0 messages received with wrong TTL
        0 messages received with bad checksum
        0 V1/V2 membership queries received
        0 V3 membership queries received
        0 membership queries received with invalid field(s)
        0 general queries received
        0 group queries received
        0 group-source queries received
        0 group-source queries dropped
        0 membership reports received
        0 membership reports received with invalid field(s)
        0 membership reports received for groups to which we belong
        0 V3 reports received without Router Alert
        0 membership reports sent
ipsec:
        0 inbound packets violated process security policy
        0 inbound packets failed due to insufficient memory
        0 invalid inbound packets
        0 outbound packets violated process security policy
        0 outbound packets with no SA available
        0 outbound packets failed due to insufficient memory
        0 outbound packets with no route available
        0 invalid outbound packets
        0 outbound packets with bundled SAs
        0 spd cache hits
        0 spd cache misses
        0 clusters copied during clone
        0 mbufs inserted during makespace
ah:
        0 packets shorter than header shows
        0 packets dropped; protocol family not supported
        0 packets dropped; no TDB
        0 packets dropped; bad KCR
        0 packets dropped; queue full
        0 packets dropped; no transform
        0 replay counter wraps
        0 packets dropped; bad authentication detected
        0 packets dropped; bad authentication length
        0 possible replay packets detected
        0 packets in
        0 packets out
        0 packets dropped; invalid TDB
        0 bytes in
        0 bytes out
        0 packets dropped; larger than IP_MAXPACKET
        0 packets blocked due to policy
        0 crypto processing failures
        0 tunnel sanity check failures
esp:
        0 packets shorter than header shows
        0 packets dropped; protocol family not supported
        0 packets dropped; no TDB
        0 packets dropped; bad KCR
        0 packets dropped; queue full
        0 packets dropped; no transform
        0 packets dropped; bad ilen
        0 replay counter wraps
        0 packets dropped; bad encryption detected
        0 packets dropped; bad authentication detected
        0 possible replay packets detected
        0 packets in
        0 packets out
        0 packets dropped; invalid TDB
        0 bytes in
        0 bytes out
        0 packets dropped; larger than IP_MAXPACKET
        0 packets blocked due to policy
        0 crypto processing failures
        0 tunnel sanity check failures
ipcomp:
        0 packets shorter than header shows
        0 packets dropped; protocol family not supported
        0 packets dropped; no TDB
        0 packets dropped; bad KCR
        0 packets dropped; queue full
        0 packets dropped; no transform
        0 replay counter wraps
        0 packets in
        0 packets out
        0 packets dropped; invalid TDB
        0 bytes in
        0 bytes out
        0 packets dropped; larger than IP_MAXPACKET
        0 packets blocked due to policy
        0 crypto processing failures
        0 packets sent uncompressed; size < compr. algo. threshold
        0 packets sent uncompressed; compression was useless
arp:
        35 ARP requests sent
        2 ARP replies sent
        2 ARP requests received
        11 ARP replies received
        13 ARP packets received
        1 total packet dropped due to no ARP entry
        4 ARP entrys timed out
        0 Duplicate IPs seen
ip6:
        21 total packets received
        0 with size smaller than minimum
        0 with data size < data length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 fragments that exceeded limit
        0 packets reassembled ok
        19 packets for this host
        0 packets forwarded
        0 packets not forwardable
        0 redirects sent
        59 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 packets that violated scope rules
        0 multicast packets which we don't join
        Input histogram:
                TCP: 2
                UDP: 10
                ICMP6: 9
        Mbuf statistics:
                11 one mbuf
                10 one ext mbuf
                0 two or more ext mbuf
        0 packets whose headers are not contiguous
        0 tunneling packets that can't find gif
        0 packets discarded because of too many headers
        3 failures of source address selection
        source addresses on an outgoing I/F
                2 link-locals
                13 globals
        source addresses on a non-outgoing I/F
                3 addresses scope=0xf
        source addresses of same scope
                2 link-locals
                3 globals
        source addresses of a different scope
                10 globals
        Source addresses selection rule applied:
                15 first candidate
                3 same address
                3 appropriate scope
icmp6:
        0 calls to icmp6_error
        0 errors not generated in response to an icmp6 message
        0 errors not generated because of rate limitation
        Output histogram:
                echo: 3
                echo reply: 3
                neighbor solicitation: 29
                MLDv2 listener report: 6
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        Input histogram:
                echo: 3
                echo reply: 3
                neighbor advertisement: 3
        Histogram of error messages to be generated:
                0 no route
                0 administratively prohibited
                0 beyond scope
                0 address unreachable
                0 port unreachable
                0 packet too big
                0 time exceed transit
                0 time exceed reassembly
                0 erroneous header field
                0 unrecognized next header
                0 unrecognized option
                0 redirect
                0 unknown
        3 message responses generated
        0 messages with too many ND options
        0 messages with bad ND options
        0 bad neighbor solicitation messages
        0 bad neighbor advertisement messages
        0 bad router solicitation messages
        0 bad router advertisement messages
        0 bad redirect messages
        0 path MTU changes
ipsec6:
        0 inbound packets violated process security policy
        0 inbound packets failed due to insufficient memory
        0 invalid inbound packets
        0 outbound packets violated process security policy
        0 outbound packets with no SA available
        0 outbound packets failed due to insufficient memory
        0 outbound packets with no route available
        0 invalid outbound packets
        0 outbound packets with bundled SAs
        0 spd cache hits
        0 spd cache misses
        0 clusters copied during clone
        0 mbufs inserted during makespace
rip6:
        0 messages received
        0 checksum calculations on inbound
        0 messages with bad checksum
        0 messages dropped due to no socket
        0 multicast messages dropped due to no socket
        0 messages dropped due to full socket buffers
        0 delivered
        0 datagrams output
```


----------

