Facebook goes down in widespread outage today
Facebook, Instagram, Whatsapp and Facebook Messenger suffered a serious disruption of the company’s services.
Downdetector outage website noticed a spike in user reports regarding services, sharing backend infrastructure, at around 2.45am
“We are aware that some people have problems accessing our apps and products,” Facebook said on Twitter.
“We are working to get things back to normal as soon as possible and we apologize for the inconvenience.”
On Twitter, Andy Stone, director of communications policy for Facebook, said the company is aware of the reported issues and is “working to get things back to normal as soon as possible.”
However, multiple security experts pointed to a Domain Name System problem as a possible culprit.
The company did not say what might be causing the outage.
A look at Down Detector (or your Twitter feed) reveals that problems are widespread. While it’s not clear exactly why the platforms are unreachable for so many people, their DNS records show that, like last week’s Slack outage, the problem is apparently DNS (it’s always DNS).
This is likely a symptom of the actual issue, and that’s that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).
There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centres due to pandemic measures.
Error – DNS_PROBE_FINISHED_NXDOMAIN
Its seems the DNS record a.ns.facebook.com is not resolvable.
The domain name system is an integral part of driving traffic on the Internet. DNS translates an address like “facebook.com” to an IP address like 69.63.176.13
If Facebook’s DNS records were gone, no one could find it.
If these DNS records have somehow been withdrawn. It could take up to 48 hours for replacement records to propagate completely on global DNS servers.
Some users in Australia have reported they can still access Facebook. However this could simply be due to certain ISPs cached DNS records and the TTL (Time to Live) attribute set for the cache expiry.
These users may also just be seeing locally cached copies of social media sites.
An interesting note. The DNS system converts names into IP address. However, Using Facebooks main IP addresses the site still can’t be accessed.
Reach Facebook by IP Address
Below are some of the most common active IP addresses for Facebook.com:
- 69.63.176.13
- 69.63.181.15
- 69.63.184.142
- 69.63.187.17
- 69.63.187.18
- 69.63.187.19
- 69.63.181.11
- 69.63.181.12
Cloudflare Senior Vice President Dane Knecht notes that Facebook’s border gateway protocol routes – BGP helps networks choose the best path to deliver Internet traffic – have been “taken off the Internet”. While some have speculated about hackers, or an internal protest about last night’s whistleblower report, there is no information yet to suggest that something malicious is to blame.
Facebook’s own site would not load at all; Instagram and WhatsApp were accessible, but could not load new content or send messages.
Instagram.com is displaying a 5xx server error message, while Facebook is producing a DNS issue message.
The problem also appears to affect its virtual reality arm, Oculus. Users can load games they already have installed and the browser works, but social features or installing new games don’t.
The outage is complete enough to affect Workplace from Facebook customers and, according to Jane Manchun Wong, internal Facebook sites.
On Twitter, Facebook Communications Manager Andy Stone says, “We are aware that some people have trouble accessing our apps and products.
We are working to get things back to normal as quickly as possible and we apologize for the inconveniences. any inconvenience “. Mike Schroepfer, who will step down from his position as CTO next year, tweeted
“We are having network problems and the teams are working as fast as possible to debug and restore as fast as possible.”
It is normal for websites and applications to experience outages, although a worldwide outage is rare.
Regarding internal failures, Adam Mosseri, the head of Instagram, tweeted that it looked like a “snowy day”.
Users reported that they cannot access Facebook in the US and Europe, as well as Australia.
Facebook’s internal systems used by employees also went down.
Doug Madory, director of Internet analytics at Kentik Inc., said it appears that the routes Facebook advertises on the Internet that tell the entire Internet how to get to his property are not available.
There’s still no connectivity to Facebook’s DNS servers:
> traceroute a.ns.facebook.com
traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets
1 dsldevice.attlocal.net (192.168.1.254) 0.484 ms 0.474 ms 0.422 ms
2 107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1) 1.592 ms 1.657 ms 1.607 ms
3 71.148.149.196 (71.148.149.196) 1.676 ms 1.697 ms 1.705 ms
4 12.242.105.110 (12.242.105.110) 11.446 ms 11.482 ms 11.328 ms
5 12.122.163.34 (12.122.163.34) 7.641 ms 7.668 ms 11.438 ms
6 cr83.sj2ca.ip.att.net (12.122.158.9) 4.025 ms 3.368 ms 3.394 ms
7 * * *Facebook.com” is registered with “registrarsafe.com” as registrar. “registrarsafe.com” is unreachable because it’s using Facebook’s DNS servers and is probably a unit of Facebook. “registrarsafe.com” itself is registered with “registrarsafe.com”.
I’m not sure of all the implications of those circular dependencies, but it probably makes it harder to get things back up if the whole chain goes down. That’s also probably why we’re seeing the domain “facebook.com” for sale on domain sites. The registrar that would normally provide the ownership info is down.
Anyway, until “a.ns.facebook.com” starts working again, Facebook is dead.
