THE BORE

General => The Superdeep Borehole => Topic started by: Tasty on December 14, 2020, 07:19:05 AM

Title: December 2020 Google and YouTube Outages [12/17: Incident #3]
Post by: Tasty on December 14, 2020, 07:19:05 AM: Down for everyone or just you?

It's not just you.

Protip: use Incognito, the issue has something to do with authentication.
Title: Re: Early Monday Google/YouTube outage
Post by: EchoRin on December 14, 2020, 07:21:08 AM: App on android is down too.
Title: Re: Early Monday Google/YouTube outage
Post by: Tasty on December 14, 2020, 07:25:18 AM: It's every Google property except search. Haven't seen a Google outage this bad in a while, if ever. Gmail is bouncing emails from other providers -- which is Very Bad™. Services relying on GCP are down too, including NYT and Pokemon Go.

And signed-in YouTube being broken is probably a bigger deal than all of that.

Someone's getting fired for a broken deploy or something lol.
Title: Re: Early Monday Google/YouTube outage
Post by: Tasty on December 14, 2020, 07:27:19 AM: Wow, what are they thinking??? Have they totally lost it???

"Google Shuts Down Gmail For Two Hours To Show Its Immense Power" (https://entertainment.theonion.com/google-shuts-down-gmail-for-two-hours-to-show-its-immen-1819595180)

spoiler (click to show/hide)
;)
[close]
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: Tasty on December 14, 2020, 07:31:02 AM: Seems to be getting resolved. I'm looking forward to the post-mortem on this one.
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: samir on December 14, 2020, 07:31:17 AM: Purple monkey fixed the problem ;)
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: samir on December 14, 2020, 07:32:02 AM: (https://i.kym-cdn.com/photos/images/newsfeed/001/350/668/6e1.jpg)
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: T-Short on December 14, 2020, 07:33:58 AM: At least we get to laugh at IoT morons

https://twitter.com/joemfbrown/status/1338452107419148290
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: Tasty on December 14, 2020, 07:40:20 AM: Quote from: Hyoushi on December 14, 2020, 07:33:58 AM
At least we get to laugh at IoT morons

https://twitter.com/joemfbrown/status/1338452107419148290

:mindblown I guess I had Hue lights but they still worked like normal lightbulbs when Wifi is down...
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: nudemacusers on December 14, 2020, 07:44:25 AM: Quote from: Hyoushi on December 14, 2020, 07:33:58 AM
At least we get to laugh at IoT morons

https://twitter.com/joemfbrown/status/1338452107419148290
Lol at this dummy, caring if his baby sits in the dark :fabulous
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: Supermanisdead on December 14, 2020, 08:31:24 AM: babies can see in the dark
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: Nintex on December 14, 2020, 09:09:35 AM: Can't wait for Google AI to drive cars, fly planes and control military aircraft :rash
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: Joe Molotov on December 14, 2020, 09:17:25 AM: Quote from: Supermanisdead on December 14, 2020, 08:31:24 AM
babies can see in the dark

I could still beat one in a fight. :steiner
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: Tasty on December 14, 2020, 11:18:20 AM: Quote from: Nintex on December 14, 2020, 09:09:35 AM
Can't wait for Google AI to drive cars, fly planes and control military aircraft :rash

Differential privacy potentially offers the best of both worlds: aggregated data parsed at cloud-scale, with the private data itself never leaving the local system and computations happening locally as well.

Google Translate used to require a cloud connection. Now it does not.

Google Maps used to require a cloud connection. Now it does not.

All of Google Photo's magic secret search sauce? (Search "sky" and you get skies, search "whatever" and get OCR search for "whatever", etc.) It wouldn't surprise me if Google is testing moving those computations locally sometime soon.

As machines get faster, it'll be more agreeable to miniaturize the cloud for every local machine rather than enforcing everyone to be a thin client.
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: Tasty on December 14, 2020, 11:21:29 AM: Where you get really fucked is YouTube, cause there ain't really a way to do distributed video hosting yet.

If like the BitTorrent protocol was rejiggered a bit there might be something there, but 0 seed videos would have to be dealt with.
Title: Re: Early Monday Google/YouTube outage E- resolved?
Post by: Tasty on December 14, 2020, 11:23:35 AM: Quote
drive cars

Already happening; Waymo is best in class and second place is way, way down.

Quote
fly planes

737 Max makes me think we'll be keeping human pilots around until I'm long dead.

Quote
and control military aircraft

They were in this game already (Boston Dynamics) and sold it off cause it made employees uncomfortable (my speculation.)
Title: Re: Early Monday Google/YouTube outage [Edit- resolved]
Post by: Nintex on December 14, 2020, 06:20:24 PM: 2045
"Alright Kris welcome to Google NUKE, so we have a test server and a live server. When you deploy make sure you deploy on staging. Never on production, one mistake could start a countdown"
"Ok... so the stage.. I got it I think..."
"To be honest I would prefer an engineer with more experience Kris but you know, budget cuts. President AOC really hates this stuff I suppose. Also because we're dealing with government contracts here this isn't the most popular department. No offense bud"
"Yeah, I'm just happy I landed this job. Beats cleaning Hooters robots lol"
"So anyway your job is simple, simulate a countdown. Really any developer can do it."
*1 hour later*
"So uh it's like live on stage right?"
"What's that Kris?"

[...]
Title: Re: Early Monday Google/YouTube outage [Edit- resolved]
Post by: Tasty on December 14, 2020, 07:13:51 PM: Google.com (2045, dehologramized)

(https://i.imgur.com/XbpsX8Q.jpg)
Title: Re: Early Monday Google/YouTube outage [Edit- resolved]
Post by: Tasty on December 15, 2020, 08:23:49 AM: https://status.cloud.google.com/incident/zall/20013

Preliminary overview of the incident.

Nothing we don't really know (auth fell over), but I hope they take a good look at this and try to improve it for next time:

Quote
Many of our internal users and tools experienced similar errors, which added delays to our outage external communication.
Title: Re: Early Monday Google/YouTube outage [Edit 2- Gmail fell over]
Post by: Tasty on December 15, 2020, 08:00:38 PM: https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8b67908fadee664c68c240ff9f529ab

Gmail died for a lot of people for about 2 hours today (I don't think I was affected.)

I posted this about Monday's incident:

Quote
Gmail is bouncing emails from other providers -- which is Very Bad™.

That actually wasn't nearly as bad as I thought, email providers are used to servers being flaky and have generous retry periods.

However, today's Gmail outage really was Very Bad™.

Quote
"Type: Permanent; SubType: General; Code: smtp; 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 https://support.google.com/mail/?p=NoSuchUser y128si147264pfg.177 - gsmtp"

Hacker News comment:

Quote
This is pretty much the worst response possible. Hard bounces mean that email delivery services are going to start automatically removing, or at least stopping delivery to, entire slews of email addresses.
A lot of clean up is going to be needed as a result of this.

To add some more details, when using a 3rd party email delivery service, those services will either black-list or just outright remove email addresses when they get a hard bounce "email address no longer exists" message back.

Some providers make re-adding an address after a hard bounce a non-trivial task, since after all, the authority on that email address just said it doesn't exist.

This is going to be really ugly.

It may even have lasted way longer and affected more users than Google is sharing:

Quote
CEO of an email marketing platform here (EmailOctopus). If anyone's curious, here's a chart showing our bounce rate to Gmail addresses over the course of the week:

https://pbs.twimg.com/media/EpUE20UXYAEa_Uv?format=jpg&name=4096x4096

That's a peak of 90% of Gmail inboxes bouncing – and this has been going on for almost 24 hours.

But it's been "resolved"... "for now"... (If you believe Google.)
Title: Re: Early Monday Google/YouTube outage [Edit 2- Gmail fell over]
Post by: Nintex on December 15, 2020, 08:07:01 PM: I didn't notice any outage today but that is indeed extremely bad.
Hard bounces are the worst case scenario and there's no easy way to fix it either. :doge

Email in general sucks so much. Hasn't changed since 1989 and all the window dressing from Microsoft and Google makes people think that it works just as reliable and easy as their other apps.

I also got an email from Google that they are going to change their policies.
Every account will be deleted after being inactive for 2 years. Now that's going to be fun for people that have linked their Outlook to Gmail through IMAP and never bother to login on Google.
I know for a fact that on the resellers console those accounts show-up as inactive. :notlikethis

Google's MO with this sort of stuff when it blows up in their face like this is usually to cut and run.
I always got the sense that after they got caught spying and crawling inboxes to serve ads and were prevented from doing so by new EU privacy laws the mail service would eventually be cast aside.
Google Apps/Suite/Business Suite/Work had its 5th rebrand in like 2 years and the partnership program was overhauled a dozen times only to revert back to what it originally was 3 years ago.
Title: Re: Early Monday Google/YouTube outage [Edit 2- Gmail fell over]
Post by: Tasty on December 15, 2020, 08:34:12 PM: Quote
I always got the sense that after they got caught spying and crawling inboxes to serve ads and were prevented from doing so by new EU privacy laws the mail service would eventually be cast aside.

:mindblown

What?

Google was always up-front about programmatic ads in Gmail, in fact it was the main element they were nervous about when launching the product ("are people OK with having their emails scanned?")

Turns out if you offer enough storage and features, the answer is pretty much yes.

Now, Google never had such ads in G Suite/Workspace Gmail. A few years back when they revamped the G Suite org (around the time Google Voice was transferred to the unit and got its first new update in years), Google decided to make consumer Gmail consistent with the enterprise version and removed programmatic ads for normal users.

Google is also trying to diversify their revenue streams away from advertising. You see this with Gmail, Calendar, Voice, Meet, etc. all now being enterprise-driven development by virtue of those products existing under the G Suite/Workspace org. You also see this with Google One and Google ending unlimited free uploads to Google Photos -- they want people to pay for storage. They want people to pay for YouTube. They don't want to be so dependent on advertising.

You could say they saw the writing on the wall, but I still see it as a proactive move and not a reactive one to EU privacy laws.

Quote
Google Cloud SVP Diane Greene said in a blog post that more than 3 million companies pay to use Google as their email provider. Google has never used these corporate accounts for ad personalization. By phasing out the practice for free Gmail accounts, Google is likely looking to ease any remaining concerns from prospective G Suite customers.

https://variety.com/2017/digital/news/google-gmail-ads-emails-1202477321/
Title: Re: Early Monday Google/YouTube outage [Edit 2- Gmail fell over]
Post by: Tasty on December 16, 2020, 08:35:48 AM: Monday's Google outage caused by critical system running out of storage (https://www.bleepingcomputer.com/news/google/google-outage-caused-by-critical-system-running-out-of-storage/)

Also yesterday's Gmail bug bit my dad, an email I sent to him last night failed. :( Hope he didn't miss anything important.
Title: Re: Early Monday Google/YouTube outage [Edit 2- Gmail fell over]
Post by: Nintex on December 16, 2020, 06:51:13 PM: Quote from: Tasty on December 16, 2020, 08:35:48 AM
Monday's Google outage caused by critical system running out of storage (https://www.bleepingcomputer.com/news/google/google-outage-caused-by-critical-system-running-out-of-storage/)

Also yesterday's Gmail bug bit my dad, an email I sent to him last night failed. :( Hope he didn't miss anything important.
Sounds like someone made a backup of Pornhub on Google Drive :heh
Title: Re: Early Monday Google/YouTube outage [Edit 3- YouTube having issues?]
Post by: Tasty on December 17, 2020, 09:54:25 AM: Having problems staying signed into YouTube.

This has certainly been a terrible, horrible, no good, very bad week for Google.
Title: Re: December 2020 Google/YouTube Outages [12/17: YouTube/Gmail, incident #3]
Post by: Tasty on December 17, 2020, 10:40:28 AM: https://downdetector.com

(https://i.imgur.com/ugfM1JQ.png)

(https://i.imgur.com/b3Pm7v7.png)
Title: Re: December 2020 Google and YouTube Outages [12/17: Incident #3]
Post by: james on December 17, 2020, 11:30:19 AM: People still use Google?