Apple Services Experiencing Outage
132 points
3 months ago
| 14 comments
| apple.com
| HN
gnarlouse
3 months ago
[-]
I don’t think I’ve every heard of an Apple Services outage until today.
reply
jonas21
3 months ago
[-]
reply
rtkwe
3 months ago
[-]
Do they offer many services for *OS apps or are they mostly independent and only service their own first party apps?
reply
ben_w
3 months ago
[-]
Theoretically they offer quite a few; in practice, I think nowhere or almost nowhere I've ever worked has gone for those options (except my own self-published games), instead preferring a custom solution that can be shared with web and android.
reply
cr125rider
3 months ago
[-]
iCloud is the backend for basically any settings/file sync for normal apps.
reply
paxys
3 months ago
[-]
But it is mostly used for periodic backup/sync. So you aren't going to really notice unless it is a very long outage.
reply
amw-zero
3 months ago
[-]
Half of the world uses Apple products, how is that not the maximum possible impact?
reply
NicoJuicy
3 months ago
[-]
No website uses Apple

A sync issue in drive would barely be noticed.

reply
x13
3 months ago
[-]
Apple Pay, with 65M users, can rely on external content from Apple.

https://developer.apple.com/documentation/applepayontheweb/d...

reply
NicoJuicy
2 months ago
[-]
That's not a website
reply
jerlam
3 months ago
[-]
You'd probably be affected if you used Sign in with Apple, same as if you used third party sign in with Google, Facebook, Microsoft, or others and their authentication was down.
reply
NicoJuicy
2 months ago
[-]
Sure, I don't use/have it though.
reply
Ferret7446
3 months ago
[-]
Because the consumer market is way less demanding than businesses. Hence why B2B and B2C are categorized separately
reply
esseph
3 months ago
[-]
Half the world? Not even remotely close. Not for MacBooks or iPhones or anything else. They're a niche player.
reply
TiredOfLife
3 months ago
[-]
Apple market share is about 30%
reply
fcsp
3 months ago
[-]
For mobile - For desktop computers, below 10% even. Given that only 73% of world population is estimated to have internet access in total, that makes it just a small fraction of world population overall
reply
kstrauser
3 months ago
[-]
As a portion of Internet users who regularly pay for and depend on hosted services, I bet that percentage is much, much higher.
reply
apparent
3 months ago
[-]
US marketshare is higher.
reply
dijit
3 months ago
[-]
To be perfectly honest with you, I wanted to come here to say that their services (on a good day) aren't very reliable anyway.

I routinely have email issues, file transfer issues (to icloud) and issues accessing their binary notarisation service.

The only thing that works routinely well is Apple Pay, however I think that it's refreshing a key lazily in the background and does not actually need a network connection to work. Good design at least.

So when I saw that they're having an outage, I thought. "All at once this time I guess".

I'll be really open here and say that I applied for an SRE job there out of hatred because whoever is in charge of SRE/Infrastructure Operations at Apple is doing a terrible job (or has terrible circumstances).

reply
eek2121
3 months ago
[-]
I've never had issues at all. I use their consumer offerings (iCloud, Apple Music, Apple TV+, Apple Arcade, etc.) and I've never personally witnessed an outage or problem. I regularly transfer stuff to iCloud from my PC running Linux, and I use the email service with a custom domain that used to live on Google until Google canceled the free tier of small business and also raised prices on the paid tier.
reply
dijit
3 months ago
[-]
It’s a bit hidden, for example if Apple Music isn’t working it will act as if my device is having an issue with the app playing; it doesn’t look like its a service problem because it hangs and freezes the UI for play- and pretty often I will force my mail to refresh and it will say the imap server is unreachable.

Regardless, these kinds of things tend to be somewhat regional. I’m based in Sweden.

reply
fsflover
3 months ago
[-]
macOS unable to open any non-Apple application (twitter.com/lapcatsoftware)

2603 points by mattsolle on Nov 12, 2020 | 1292 comments

> I am currently unable to work because macOS sends hashes of every opened executable to some server of theirs and [it's down]

https://news.ycombinator.com/item?id=25074959

reply
1970-01-01
3 months ago
[-]
Even downdetector.com has nothing about Apple Services being offline: https://downdetector.com/search/?q=apple
reply
arprocter
3 months ago
[-]
reply
1970-01-01
3 months ago
[-]
They're in double-digit numbers for complaints. Must be a new record :)
reply
g947o
3 months ago
[-]
reply
crims0n
3 months ago
[-]
Same... to their credit, even during the major cloud outages I don't recall Apple services going down.
reply
dabbz
3 months ago
[-]
If I remember right they primarily use Google Cloud with a small mix of multi-cloud stuff for iCloud
reply
8fingerlouie
3 months ago
[-]
depends on where in the world you live.

If you live in a region where they operate their own data centers, you will be running on Apple data centers. If not, you're running on a mix of Google Cloud and AWS (IIRC). They used to use Azure as well, but I think that's no longer the case.

In any case, your data is encrypted (by Apple) before being uploaded to Google or AWS, and only Apple has that key. Whatever E2EE encryption you use will be applied on top of that.

reply
dabbz
3 months ago
[-]
Last I checked they were phasing out their own DCs in favor of cloud-provided services. Though it's been a while since I have heard anything about it, so maybe those plans got canceled. It could have also been phasing out those DCs for only the specific services and not all services. My details on the whole thing are fuzzy at best.
reply
8fingerlouie
3 months ago
[-]
As far as I know, everything iCloud and Apple Intelligence runs off of their own data centers if you happen to live "near" one, but you could still be using AWS and/or Google as well.

I live near the Danish Apple data center, and pretty much all my iCloud traffic goes there, with a small fraction (<10%) going to Stockholm, which has both AWS and Google data centers, so I assume they're using both for geographical redundancy (erasure coding)

It gets a bit more fuzzy once you start moving into Movies/Music/TV/Billing/whatever as well as their backend services for the store and monitoring.

reply
dabbz
3 months ago
[-]
Got it, yea my inside sources were as they relate to Siri and that was a decade ago at this point.
reply
bombcar
3 months ago
[-]
If the outages only happen to services that aren’t used much I understand why.
reply
Etheryte
3 months ago
[-]
Apple is one of the largest service providers worldwide. Services are their second largest revenue stream after the iPhone.
reply
justapassenger
3 months ago
[-]
1. About 25% of their service revenue is from charging commissions in app store and other 25% of the revenue is Google paying them for search default. Other services include things like insurance (applecare) That's not exactly same type services that most of the people would be thinking about.

2. A lot of their services have less criticality (and it's not a ding at them - it's often very explicit design choice).

3. App store having hiccups or iCloud backups being delayed it's not something that will usually gather enough attention of media.

reply
ctime
3 months ago
[-]
You might be amazed to know how critical Services are to functioning Apple devices. While they mostly can run offline, there are dozens and dozens of services that Apple runs that modern ecosystems require (like certificate related stuff). Other oddball things related to iCloud, APNS and the private services like iCloud relay are all extremely critical to billions of devices. Thankfully the all mostly fail open (captive portal is particularly tricky). Not saying they are as critical or visible as, say, Google.com going down, but none the less would have a very very large and visible problem if they all did go down suddenly. Thankfully, due to Apple design philosophy, most are totally decentralized and teams are given almost complete autonomy on how services are ran, which makes them a huge confusing mess but also, kind of a feature as Apple generally expects them all to fail in odd ways and the software can generally handle it.
reply
amw-zero
3 months ago
[-]
that's a fantastic point.
reply
giancarlostoro
3 months ago
[-]
Seems both TV and Music are affected, I wonder if it's some storage service somewhere just totally degraded. I love CloudFlare for their total transparency reports about outages. I wish every other company would follow the same standard.
reply
temp0826
3 months ago
[-]
I had to try a few times to download from the app store this morning. Thought it was a dns/blocking issue on my part.
reply
thoughtpalette
3 months ago
[-]
Been using Apple Music all day today, streaming. Have yet to encounter an issue.
reply
giancarlostoro
3 months ago
[-]
Must be a partial outage then? I was using Music earlier today as well. On my phone I download most of my music so I might never notice an outage though.
reply
thoughtpalette
3 months ago
[-]
Good point! I might not have noticed.
reply
andersco
3 months ago
[-]
Also seeing outages relating to App Store Connect, currently unable to submit builds to TestFlight.
reply
snarf21
3 months ago
[-]
The ASC website isn't loading correctly either.
reply
sneak
3 months ago
[-]
Apple services are realtime keys to:

- activation: wiping an iOS/iPadOS/visionOS device, or reinstalling a mac after a full disk wipe

- apns: push notifications, used for realtime notifications for all apps on iOS/iPadOS, even things like Signal that do not use Apple's messaging infrastructure

- imessage: enough said

- boot ticket signing: required for any mac to do an OS update (it's serialized to the CPU's ECID)

any one of these going down for any significant period of time is going to cause widespread global economic disruption.

reply
drnick1
3 months ago
[-]
And this is precisely why you need decentralized systems. It's absurd that text messages are routed through Apple's servers, or that you need Apple's permission to wipe a device.
reply
NetMageSCW
3 months ago
[-]
You need Apple’s permission to wipe a device so when you steal a device, you can’t wipe it and sell it. It’s part of Apple’s ongoing efforts to prevent stolen devices providing any value when stolen.
reply
fsflover
3 months ago
[-]
If this is mandatory, then it's not really your device.
reply
Ferret7446
3 months ago
[-]
Well, the alternative is that it's not your device either when someone steals it, it's now their device. The average user cares far more about that
reply
fsflover
3 months ago
[-]
If you can even opt-out of that, this is basically a nanny device.
reply
FridgeSeal
3 months ago
[-]
> It's absurd that text messages are routed through Apple's servers

Well, that’s because…they’re not text messages.

reply
gjsman-1000
3 months ago
[-]
If they were text messages, it's absurd that text messages are often routed through two servers even!
reply
sosodev
3 months ago
[-]
Aren't all text messages routed through a server? I guess it's more decentralized when it comes to telecom though.
reply
joecool1029
3 months ago
[-]
> Aren't all text messages routed through a server?

For SMS, no not a central server for all carriers. When SMS service originally launched it wasn’t even cross-carrier.

MMS acts like a mailserver at each carrier, sends a push notification and your device has like a day to download full message.

For RCS it’s supposed to be federated but carriers gave up and it’s now centralized through Google Jibe.

reply
NetMageSCW
3 months ago
[-]
I don’t think it’s any more decentralized than Apple if you are talking about MMS (instead of SMS). Your carrier still provides the MMS servers.
reply
apparent
3 months ago
[-]
Did this affect notifications/app updates? I found that podcasts from my third-party app were not showing up for me when they normally do. I would assume that when I manually refresh the list, that wouldn't rely on Apple servers, but given this outage maybe there's a linkage?
reply
jordemort
3 months ago
[-]
Sorry, it's my fault, I migrated my playlists over to TIDAL earlier today
reply
resonious
3 months ago
[-]
I wonder about APNs and Apple Business Manager. I've heard from people seeing weird stuff happening on those products but I don't see it on the report here.
reply
niwtsol
3 months ago
[-]
This outage was affecting developer.apple.com and pushing iOS app versions to test flight/production. I thought I was going crazy, thanks for posting this.
reply
joecool1029
3 months ago
[-]
Curious to see the Testflight/Support apps updated just after. I’m betting they pushed an upgrade to something in their services and it was bumpy.
reply
impure
3 months ago
[-]
I'm getting a lot of 500 errors when trying to upload my app to App Store Connect. Also the website is loading slowly.
reply
binsquare
3 months ago
[-]
Failed to install/download newest xcode after updating Mac.

Guess this explains it.

Does Apple internally do something like COE's like Amazon?

reply
rvz
3 months ago
[-]
Looking forward to the post-mortem
reply
bitpush
3 months ago
[-]
I'd be really surprised if Apple were forthcoming. Apple famously holds all cards close to the chest, so I dont expect anything from them. Happy to be wrong though.
reply
seper8
3 months ago
[-]
Noticed this too earlier today. App Store apps not downloading.
reply
JSR_FDED
3 months ago
[-]
Everything is working 100% for me
reply