that will be good to be a part of!

would thoroughly recommend this! 😄 The beer and chat seems to always flow easy with @bryan.finster486 as a side note also keep an eye out in #happy-hour on future DOES orientated community events in addition to the BVSSH community get togethers! - get the best of both worlds! #Dockside #rebellionTogether

Always a good time when the Funsters host… oops I mean Finsters 😊

LOL. A friend of mine at Walmart gave me that nickname when I joined Platform. 😂

Yay? ;-)

Someone needs a pipeline for DNS configuration changes?

I find it hard to believe that FB, which supposedly has a very strong eng culture, does not have a pipeline for config changes. Possibly so?

Facebook's border gateway protocol routes — BGP helps networks pick the best path to deliver internet traffic — have been "withdrawn from the internet."

I hope they publish lessons learned just like AWS

Haha yea, it would be great if they can be forthcoming about the information they learned from their internal research...

@dvancouvering it's a dumpster fire everywhere.

Everywhere meaning at every company?

Basically.

Yea, it's a great point, thanks Bryan

I find it both horrifying and strangely comforting. :)

lol

But improvement is a growth industry.

I was listening to an old episode of Soft Skills Engineering (a Bryan rec) and they said they don't know of anyone that worked somewhere that was truly happy with where their code was. Helped me set expectations a little better

When we created the Walmart CD platform. The Sr. Director (ex-FB) was helping them get better at CD using our tools, so…

Having seen dozens of companies and hundreds of teams: Every organisation, big or small, is always messy.... I mean: Has a complex system of problems. There seem to be inherent risks in the attempt of trying to build an organisation out of people that is somewhat doomed to fail... as well as create prosperity and good things as long as the failing is failing to be complete failing. So, apparently Facebook as a system of humans is incapable of 100% availability over long times.

I saw a report that their badge system went down too.

Their domain name appears to have been marked for sale according to domain services? My suspicions of foul play increase... https://www.wtrf.com/news/national-news/facebook-listed-for-sale-after-website-goes-down/

https://blog.cloudflare.com/october-2021-facebook-outage/

i saw where there was a reddit user who claimed to be working on the recovery team and posted in the sysadmin subreddit about the outage. they appear to have deleted their account though.

Are there articles about how a company like facebook goes about the Build vs Buy of using their own data center vs a cloud?

Thoughts are definitely for the Facebook engineers today! A real test of psychological safety when you bring down your global network

Seems like a BGP routing change? Hard to put into a pipeline, I suppose. But, it seems strange that one would change all routes at the same time. It would be great to understand this better. Of course, these changes can be made by anyone. Here were two I remember impacting AWS and Google: https://www.thousandeyes.com/blog/route-leak-causes-amazon-and-aws-outage, https://www.cnet.com/news/how-pakistan-knocked-youtube-offline-and-how-to-make-sure-it-never-happens-again/

There is the equivalent SRE discipline of network reliability engineering taking on maturing software network automation with all the good devops practices. Maybe a good topic for the next DOES! But not really new either since carrier networks have been 'software driven' highly distributed, high reliability environment with powerful control planes based on slow changing internet standards. Some other high profile outages examples with BGP flowspec (CenturyLink and cloudflare). We've all been there having a bad day (perhaps with a less dramatic impact on market cap)! Facebook provided a general official update before: https://engineering.fb.com/2021/10/04/networking-traffic/outage/ The transparency on the RFO will be interesting.