December 2020 Google and YouTube Outages [12/17: Incident #3]

[Tasty]

Down for everyone or just you?

It's not just you.

Protip: use Incognito, the issue has something to do with authentication.

[EchoRin]

App on android is down too.

[Tasty]

It's every Google property except search. Haven't seen a Google outage this bad in a while, if ever. Gmail is bouncing emails from other providers -- which is Very Bad™. Services relying on GCP  are down too, including NYT and Pokemon Go.

And signed-in YouTube being broken is probably a bigger deal than all of that.

Someone's getting fired for a broken deploy or something lol.

[Tasty]

Wow, what are they thinking??? Have they totally lost it???

"Google Shuts Down Gmail For Two Hours To Show Its Immense Power"

spoiler (click to show/hide)

[close]

[Tasty]

Seems to be getting resolved. I'm looking forward to the post-mortem on this one.

[samir]

Purple monkey fixed the problem 

[samir]

[T-Short]

At least we get to laugh at IoT morons

https://twitter.com/joemfbrown/status/1338452107419148290

[Tasty]

At least we get to laugh at IoT morons

https://twitter.com/joemfbrown/status/1338452107419148290

I guess I had Hue lights but they still worked like normal lightbulbs when Wifi is down...

[nudemacusers]

At least we get to laugh at IoT morons

https://twitter.com/joemfbrown/status/1338452107419148290

Lol at this dummy, caring if his baby sits in the dark

[Supermanisdead]

babies can see in the dark

[Nintex]

Can't wait for Google AI to drive cars, fly planes and control military aircraft

[Joe Molotov]

babies can see in the dark

I could still beat one in a fight.

[Tasty]

Can't wait for Google AI to drive cars, fly planes and control military aircraft

Differential privacy potentially offers the best of both worlds: aggregated data parsed at cloud-scale, with the private data itself never leaving the local system and computations happening locally as well.

Google Translate used to require a cloud connection. Now it does not.

Google Maps used to require a cloud connection. Now it does not.

All of Google Photo's magic secret search sauce? (Search "sky" and you get skies, search "whatever" and get OCR search for "whatever", etc.) It wouldn't surprise me if Google is testing moving those computations locally sometime soon.

As machines get faster, it'll be more agreeable to miniaturize the cloud for every local machine rather than enforcing everyone to be a thin client.

[Tasty]

Where you get really fucked is YouTube, cause there ain't really a way to do distributed video hosting yet.

If like the BitTorrent protocol was rejiggered a bit there might be something there, but 0 seed videos would have to be dealt with.

[Tasty]

drive cars

Already happening; Waymo is best in class and second place is way, way down.

fly planes

737 Max makes me think we'll be keeping human pilots around until I'm long dead.

and control military aircraft

They were in this game already (Boston Dynamics) and sold it off cause it made employees uncomfortable (my speculation.)

[Nintex]

2045
"Alright Kris welcome to Google NUKE, so we have a test server and a live server. When you deploy make sure you deploy on staging. Never on production, one mistake could start a countdown"
"Ok... so the stage.. I got it I think..."
"To be honest I would prefer an engineer with more experience Kris but you know, budget cuts. President AOC really hates this stuff I suppose. Also because we're dealing with government contracts here this isn't the most popular department. No offense bud"
"Yeah, I'm just happy I landed this job. Beats cleaning Hooters robots lol"
"So anyway your job is simple, simulate a countdown. Really any developer can do it."
*1 hour later*
"So uh it's like live on stage right?"
"What's that Kris?"

[...]

[Tasty]

Google.com (2045, dehologramized)

[Tasty]

https://status.cloud.google.com/incident/zall/20013

Preliminary overview of the incident.

Nothing we don't really know (auth fell over), but I hope they take a good look at this and try to improve it for next time:

Many of our internal users and tools experienced similar errors, which added delays to our outage external communication.

[Tasty]

https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8b67908fadee664c68c240ff9f529ab

Gmail died for a lot of people for about 2 hours today (I don't think I was affected.)

I posted this about Monday's incident:

Gmail is bouncing emails from other providers -- which is Very Bad™.

That actually wasn't nearly as bad as I thought, email providers are used to servers being flaky and have generous retry periods.

However, today's Gmail outage really was Very Bad™.

"Type: Permanent; SubType: General; Code: smtp; 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 https://support.google.com/mail/?p=NoSuchUser y128si147264pfg.177 - gsmtp"

Hacker News comment:

This is pretty much the worst response possible. Hard bounces mean that email delivery services are going to start automatically removing, or at least stopping delivery to, entire slews of email addresses.
A lot of clean up is going to be needed as a result of this.

To add some more details, when using a 3rd party email delivery service, those services will either black-list or just outright remove email addresses when they get a hard bounce "email address no longer exists" message back.

Some providers make re-adding an address after a hard bounce a non-trivial task, since after all, the authority on that email address just said it doesn't exist.

This is going to be really ugly.

It may even have lasted way longer and affected more users than Google is sharing:

CEO of an email marketing platform here (EmailOctopus). If anyone's curious, here's a chart showing our bounce rate to Gmail addresses over the course of the week:

https://pbs.twimg.com/media/EpUE20UXYAEa_Uv?format=jpg&name=4096x4096

That's a peak of 90% of Gmail inboxes bouncing – and this has been going on for almost 24 hours.

But it's been "resolved"... "for now"... (If you believe Google.)

[Nintex]

I didn't notice any outage today but that is indeed extremely bad.
Hard bounces are the worst case scenario and there's no easy way to fix it either. 

Email in general sucks so much. Hasn't changed since 1989 and all the window dressing from Microsoft and Google makes people think that it works just as reliable and easy as their other apps.

I also got an email from Google that they are going to change their policies.
Every account will be deleted after being inactive for 2 years. Now that's going to be fun for people that have linked their Outlook to Gmail through IMAP and never bother to login on Google.
I know for a fact that on the resellers console those accounts show-up as inactive. 

Google's MO with this sort of stuff when it blows up in their face like this is usually to cut and run.
I always got the sense that after they got caught spying and crawling inboxes to serve ads and were prevented from doing so by new EU privacy laws the mail service would eventually be cast aside.
Google Apps/Suite/Business Suite/Work had its 5th rebrand in like 2 years and the partnership program was overhauled a dozen times only to revert back to what it originally was 3 years ago.

[Tasty]

I always got the sense that after they got caught spying and crawling inboxes to serve ads and were prevented from doing so by new EU privacy laws the mail service would eventually be cast aside.

What?

Google was always up-front about programmatic ads in Gmail, in fact it was the main element they were nervous about when launching the product ("are  people OK with having their emails scanned?")

Turns out if you offer enough storage and features, the answer is pretty much yes.

Now, Google never had such ads in G Suite/Workspace Gmail. A few years back when they revamped the G Suite org (around the time Google Voice was transferred to the unit and got its first new update in years), Google decided to make consumer Gmail consistent with the enterprise version and removed programmatic ads for normal users.

Google is also trying to diversify their revenue streams away from advertising. You see this with Gmail, Calendar, Voice, Meet, etc. all now being enterprise-driven development by virtue of those products existing under the G Suite/Workspace org. You also see this with Google One and Google ending unlimited free uploads to Google Photos -- they want people to pay for storage. They want people to pay for YouTube. They don't want to be so dependent on advertising.

You could say they saw the writing on the wall, but I still see it as a proactive move and not a reactive one to EU privacy laws.

Google Cloud SVP Diane Greene said in a blog post that more than 3 million companies pay to use Google as their email provider. Google has never used these corporate accounts for ad personalization. By phasing out the practice for free Gmail accounts, Google is likely looking to ease any remaining concerns from prospective G Suite customers.

https://variety.com/2017/digital/news/google-gmail-ads-emails-1202477321/

[Tasty]

Monday's Google outage caused by critical system running out of storage



Also yesterday's Gmail bug bit my dad, an email I sent to him last night failed. Hope he didn't miss anything important.

[Nintex]

Monday's Google outage caused by critical system running out of storage



Also yesterday's Gmail bug bit my dad, an email I sent to him last night failed. Hope he didn't miss anything important.

Sounds like someone made a backup of Pornhub on Google Drive

[Tasty]

Having problems staying signed into YouTube.

This has certainly been a terrible, horrible, no good, very bad week for Google.

[Tasty]

https://downdetector.com

[james]

People still use Google?