# docs.freebsd.org down?



## frijsdijk (Jul 8, 2013)

I checked from several points on the Internet, but it seems down. Is there maintenance or are there troubles?

```
# host docs.freebsd.org
docs.freebsd.org is an alias for www.freebsd.org.
www.freebsd.org is an alias for wfe0.ysv.freebsd.org.
wfe0.ysv.freebsd.org has address 8.8.178.110
wfe0.ysv.freebsd.org has IPv6 address 2001:1900:2254:206a::50:0
wfe0.ysv.freebsd.org mail is handled by 0 .

# ping 8.8.178.110
PING 8.8.178.110 (8.8.178.110): 56 data bytes
^C
--- 8.8.178.110 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
(root@monitoring)[/usr/local/etc/nagios/servers]# telnet 8.8.178.110 80
Trying 8.8.178.110...
^C
```
It seems to affect pkg-audit as well.


----------



## Amzo (Jul 8, 2013)

I can't even access bugs at the minute. 


```
Error 118 (net::ERR_CONNECTION_TIMED_OUT): The operation timed out.
```

And I also can't access Docs as well. Is it possible the servers are down or a DDoS attack?


----------



## fonz (Jul 8, 2013)

It appears the entire website (minus these forums, fortunately) is down. I'd imagine that if there was (scheduled) maintenance being done there'd be a fallback server.


----------



## Paul-LKW (Jul 8, 2013)

*FreeBSD Main Site Down whole day*

Today is a bad day, the main site and SVN server are down whole day !


----------



## drhowarddrfine (Jul 8, 2013)

Oops


----------



## Paul-LKW (Jul 8, 2013)

It is up again ^_^


----------



## cpm@ (Jul 8, 2013)

Seems that has been addressed 

```
[CMD]% traceroute www.freebsd.org[/CMD]
traceroute to wfe0.ysv.freebsd.org (8.8.178.110), 64 hops max, 40 byte packets
 1  114.Red-80-58-67.staticIP.rima-tde.net (80.58.67.114)  43.067 ms  42.815 ms  43.083 ms
 2  * * *
 3  178.Red-80-58-84.staticIP.rima-tde.net (80.58.84.178)  59.263 ms  60.266 ms  60.109 ms
 4  Et4-0-0-1-grtmadad1.red.telefonica-wholesale.net (84.16.10.73)  58.317 ms  58.241 ms  60.057 ms
 5  Te0-3-0-1-grtlontlw1.red.telefonica-wholesale.net (84.16.12.150)  97.315 ms  90.808 ms  98.495 ms
 6  te7-3-10G.ar7.LON3.gblx.net (64.208.27.17)  83.481 ms  83.486 ms  83.001 ms
 7  ae6.scr4.LON3.gblx.net (67.17.106.150)  95.889 ms  95.228 ms  96.013 ms
 8  xe11-2-0-10G.scr3.SNV2.gblx.net (67.16.164.18)  231.164 ms  230.697 ms  231.037 ms
 9  e8-1-20G.ar5.SJC2.gblx.net (67.16.145.118)  227.316 ms
    e5-3-40G.ar5.SJC2.gblx.net (67.17.72.14)  240.309 ms  230.408 ms
10  YAHOO.TenGigabitEthernet2-4.1189.ar3.SJC2.gblx.net (208.48.239.254)  223.465 ms  224.308 ms  262.153 ms
11  bgp1-ext.ysv.freebsd.org (216.115.101.227)  222.933 ms  224.272 ms  223.869 ms
12  routerer-ext.ysv.freebsd.org (8.8.178.93)  225.616 ms  226.498 ms  225.354 ms
13  wfe0.ysv.freebsd.org (8.8.178.110)  226.122 ms  224.680 ms  224.402 ms
```

http://lists.freebsd.org/pipermail/freebsd-questions/2013-July/252080.html.


----------



## freeunix (Jul 8, 2013)

*Why was freebsd.org down a few hours ago?*

A while ago freebsd.org was not accessible. Why? It isn't my Internet connection, I tried several "is website down" services.


----------



## freeunix (Jul 8, 2013)

Paul-LKW said:
			
		

> Today is a badly day, the main site and svn server is down whole day !!



Today is a _bad(ly)_ day, the main site (except the forums) and SVNn server have been down for about four hours.


----------



## DutchDaemon (Jul 8, 2013)

As far as I can see there was a temporary routing error within the Yahoo! network where these services are located (and where the forums aren't).


----------



## fonz (Jul 8, 2013)

Two threads about the same subject merged.


----------



## peter@ (Jul 9, 2013)

drhowarddrfine said:
			
		

> Oops



Hey, that's a trade secret!

Seriously though...  here's the short version.

While I was asleep, one of our redundant uplinks went dark.  This wouldn't have been a problem because it was redundant, and we had two, right?

The problem was there was a misunderstanding about an openbgpd configuration.  That shouldn't have been a problem either except the openbgpd port doesn't do ECMP (openbgpd itself does, but it is violently incompatible with our RADIX_MPATH and using the port is effectively suicidal if you turn it on).

So, there was a lot of sudden, unexpected learning today. Along the way things managed to get Really Fixed(TM) multiple times, with the predictable outcome of more (albeit relatively short) outages.

We are now in a state where it should fail over correctly but we inflicted unspeakable atrocities to make this happen. We have sinned in the full view of the BGP gods.  The hack, while it works, is All Wrong(TM).  In retrospect it has become clear that I just Don't Understand BGP.

Tomorrow, after sleep, and within physical reach of the gear, we shall try and do it right.


----------



## wblock@ (Jul 9, 2013)

Thanks for working on this.  While not fun at the time, that kind of experience results in valuable improvements.  Maybe a scheduled (automated?) failover test?  Once a year would make it Router Appreciation Day.


----------



## peter@ (Jul 9, 2013)

We actually do a failover and back every month, at least so far as machines that we control go.  What got us this time was something upstream of our gear that lost its link and that exposed a configuration error.

We have had this happen once at ISC but in a different way.  That was an upstream switch failure.


----------

