▲madjam00237 minutes ago
[-] Looking forward to the post mortem on this one. We weren't affected (just using the CDN), and people are saying they weren't affected who are using Cloudflare Workers (a previous culprit which we've since moved off), so I wonder what service / API was actually affected that brought down multiple websites with a 500 but not all of them.
Wise was just down which is a pretty big one.
Also odd how some websites were down this time that previously weren't down with the global outage in November
reassess_blind27 minutes ago
[-] Yeah it's strange. My sites that are are proxied through Cloudflare remained up, but Supabase was taken offline so some backends were down. Either a regional PoP style issue, or a specific API or service had to be used to be affected.
reply▲da_grift_shift5 minutes ago
[-] The excuse:
>A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning.
>The change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components.
>We will share more information as we have it today.
https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q
archon81027 minutes ago
[-] Our locations excluded from Cloudflare WAF were up, but the rest was down. I think WAF took a dump.
reply▲CDN was definitely down also. We were widely impacted by it with 500's.
reply▲thinkindie33 minutes ago
[-] we were not affected too and we realised it was Cloudflare because Linear was down and they were mentioning an upstream service. Also Ecosia was affected, and I then realised they might be relying on Cloudflare too.
reply▲m_mueller29 minutes ago
[-] Maven Repository was down for me for a while, now it recovered.
reply▲gowthamgts1232 minutes ago
[-] CDN was also affected for some customers. we were down with 500.
reply▲was interesting, some of our stuff failed, but some other stuff that used cloudflare indirectly didn't.
reply▲cryptonym24 minutes ago
[-] > Looking forward to the post mortem
This is becoming a meme.
meandmycode20 minutes ago
[-] This has to be setting off some alarm bells internally, a well written postmortem on an occasional issue, great, but when your postmortem talks about learnings and improvements yet major outages keep happening, it becomes meaningless..
reply▲This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.
reply▲At least this warrants a good review of anyone's dependency on cloudflare.
If it turns out that this was really just random bad luck, it shouldn't affect their reputation (if humans were rational, that is...)
But if it is what many people seem to imply, that this is the outcome of internal problems/cuttings/restructuring/profit-increase etc, then I truly very much hope it affects their reputation.
But I'm afraid it won't. Just like Microsoft continues to push out software, that, compared to competitors, is unstable, insecure, frustrating to use, lacks features, etc, without it harming their reputation or even bottomlines too much. I'm afraid Cloudflare has a de-facto monopoly (technically: big moat) and can get away with offering poorer quality, for increasing pricing by now.
zelphirkalt12 seconds ago
[-] Microsoft's reputation couldn't be much lower at this point, that's their trick.
The issue is the uninformed masses being led to use Windows when they buy a computer. They don't even know how much better a system could work, and so they accept whatever is shoved down their throats.
coffeebeqn46 minutes ago
[-] Vibe infrastructure
reply▲So that is what the best case definition of what "Vibe Engineering" is.
reply▲MrAureliusR38 minutes ago
[-] well that's the thing, such a huge number of companies route
all their traffic through Cloudflare. This is at least partially because for a long time, there was no other company that could really do what Cloudflare does, especially not at the scales they do. As much as I despise Cloudflare as a company, their blog posts about stopping attacks and such are extremely interesting. The amount of bandwidth their network can absorb is jaw-dropping.
I've said to many people/friends that use Cloudflare to look elsewhere. When such a huge percentage of the internet flows through a single provider, and when that provider offers a service that allows them to decrypt all your traffic (if you let them install HTTPS certs for you), not only is that a hugely juicy target for nation-states but the company itself has too much power.
But again, what other companies can offer the insane amount of protection they can?
This will be another post-mortem of...config file messed...did not catch...promise to be doing better next....We are sorry.
They problem is architectural.
PlotCitizen58 minutes ago
[-] This is a good reminder for everyone to reconsider making all of their websites depend on a single centralized point of failure. There are many alternatives to the different services which Cloudflare offers.
reply▲But the nature of a CDN and most other products CF offers, is central by nature.
If you switch from CF to the next CF competitor, you've not improved this dependency.
The alternative here, is complex or even non-existing. Complex would be some system that allows you to hotswap a CDN, or to have fallback DDOS protection services, or to build you own in-house. Which, IMO, is the worst to do if your business is elsewhere. If you sell, say, petfood online, the dependency-risk that comes with a vendor like CF, quite certainly is less than the investment needed- and risk associted with- building a DDOS protection or CDN on your own; all investment that's not directed to selling more pet-food or get higher margins at doing so.
agnivade36 minutes ago
[-] You can load-balance between CDN vendors as well
reply▲Then your load balancer becomes the single point of failure.
reply▲coffeebeqn46 minutes ago
[-] We just love to merge the internet into single points of failure
reply▲This is just how free markets work, on the internet with no "physical" limitations it is simply accelerated.
Left alone corporations to rival governments emerge, which are completely unaccountable. At least there is some accountability of governments to the people, depending on your flavour of government.
mschuster912 minutes ago
[-] no one
loves the need for CDNs other than maybe video streaming services.
the problem is, below a certain scale you can't operate anything on the internet these days without hiding behind a WAF/CDN combo... with the cut-off mark being "we can afford a 24/7 ops team". even if you run a small niche forum no one cares about, all it takes is one disgruntled donghead that you ban to ruin the fun - ddos attacks are cheap and easy to get these days.
and on top of that comes the shodan skiddie crowd. some 0day pops up, chances are high someone WILL try it out in less than 60 minutes. hell, look into any web server log, the amount of blind guessing attacks (e.g. /wp-admin/..., /system/login, /user/login) or path traversal attempts is insane.
CDN/WAFs are a natural and inevitable outcome of our governments and regulatory agencies not giving a shit about internet security and punishing bad actors.
karmakurtisaani59 minutes ago
[-] Probably fired a lot of their best people in the past few years and replaced it with AI. They have a de-facto monopoly, so we'll just accept it and wait patiently until they fix the problem. You know, business as usual in the grift economy.
reply▲5d41402abc4b41 minutes ago
[-] >They have a de-facto monopoly
On what? There are lots of CDN providers out there.
They do fare more than just CDN. It's the combination of service, features, reach, price, and the integration of it all.
reply▲There's only one that lets everyone sign up for free.
reply▲The "AI agents" are on holiday when an outage like this happens.
reply▲We are now seeing which companies do not consider the third party risk of single point of failures in systems they do not control as part of their infrastructure and what their contingency plan is.
It turns out so far, there isn't one. Other than contacting the CEO of Cloudflare rather than switching on a temporary mitigation measure to ensure minimal downtime.
Therefore, many engineers at affected companies would have failed their own systems design interviews.
throwaway4234635 minutes ago
[-] Alternative infrastructure costs money, and it's hard to get approval from leadership in many cases. I think many know what the ideal solution looks like, but anything linked to budgets is often out of the engineer's hands.
In some cases it is also a valid business decision. If you have 2 hour down time every 5 years, it may not have a significant revenue impact. Most customers think it's too much bother to switch to a competitor anyway, and even if it were simple the competition might not be better. Nobody gets fired for buying IBM
The decision was probably made by someone else who moved on to a different company, so they can blame that person. It's only when down time significantly impacts your future ARR (and bonus) that leadership cares (assuming that someone can even prove that they actually lose customers).
cryptonym42 minutes ago
[-] Sometimes it's not worth it. Your plan is just to accept you'll be off for a day or two, while you switch to a competitor.
reply▲reply▲Yeah. I only work for a small company, but you can be certain we will not update the status page if only a small portion of customers are affected, and if we are fully down, rest assured there will be no available hands to keep the status page updated
reply▲>rest assured there will be no available hands to keep the status page updated
That's not how status pages if implemented correctly work. The real reason status pages aren't updated is SLAs. If you agree on a contract to have 99.99% uptime your status page better reflect that or it invalidates many contracts. This is why AWS also lies about it's uptime and status page.
These services rarely experience outages according their own figures but rather 'degraded performance' or some other language that talks around the issue rather than acknowledging it.
It's like when buying a house you need an independent surveyor not the one offered by the developer/seller to check for problems with foundations or rotting timber.
SLA’s usually just give you a small credit for the exact period of the incident, which is arymetric to the impact. We always have to negotiate for termination rights for failing to meet SLA standards but, in reality, we never exercise them.
Reality is that in an incident, everyone is focused on fixing issue, not updating status pages; automated checks fail or have false positives often too. :/
laurent12345642 minutes ago
[-] This is weird - at this level contracts are supposed to be rock solid so why wouldn't they require accurate status reporting? That's trivial to implement, and you can even require to have it on a neutral third-party like UptimeRobot and be done with it.
I'm sure there are gray areas in such contracts but something being down or not is pretty black and white.
franga200032 minutes ago
[-] > something being down or not is pretty black and white
This is so obviously not true that I'm not sure if you're even being serious.
Is the control panel being inaccessible for one region "down"? Is their DNS "down" if the edit API doesn't work, but existing records still get resolved? Is their reverse proxy service "down" if it's still proxying fine, just not caching assets?
> I'm sure there are gray areas in such contracts but something being down or not is pretty black and white.
Is it? Say you've got some big geographically distributed service doing some billions of requests per day with a background error rate of 0.0001%, what's your threshold for saying whether the service is up or down? Your error rate might go to 0.0002% because a particular customer has an issue so that customer would say it's down for them, but for all your other customers it would be working as normal.
lucianbr43 minutes ago
[-] Are the contracts so easy to bypass? Who signs a contract with an SLA knowing the service provider will just lie about the availability? Is the client supposed to sue the provider any time there is an SLA breach?
reply▲netdevphoenix38 minutes ago
[-] Anyone who doesn't have any choice financially or gnostically. Same reason why people pay Netflix despite the low quality of most of their shows and the constant termination of tv series after 1 season. Same reason why people put up with Meta not caring about moderating or harmful content. The power dynamics resemble a monopoly
reply▲Most of services are not really critical but customers want to have 99.999% on the paper.
Most of the time people will just get by and ignore even full day of downtime as minor inconvenience. Loss of revenue for the day - well you most likely will have to eat that, because going to court and having lawyers fighting over it most likely will cost you as much as just forgetting about it.
If your company goes bankrupt because AWS/Cloudflare/GCP/Azure is down for a day or two - guess what - you won't have money to sue them ¯\_(ツ)_/¯ and most likely will have bunch of more pressing problems on your hand.
The client is supposed to monitor availability themselves, that is how these contracts work.
reply▲The company that is trying to cancel its contract early needs to prove the SLA was violated, which is very easy of the company providing the service also provides a page that says their SLA was violated. Otherwise it's much harder to prove.
reply▲8cvor6j844qw_d647 minutes ago
[-] I imagine there will be many levels of "approvals" to get the status page actually showing down, since SLA uptime contracts is involved.
reply▲I work for a small company. We have no written SLA agreements.
reply▲lawnchair46 minutes ago
[-] I have to say that if an incident becomes so overwhelming that nobody can spare even a moment to communicate with customers, that points to a deeper operational problem. A status page is not something you update only when things are calm. It is part of the response itself. It is how you keep users informed and maintain trust when everything else is going wrong.
If communication disappears entirely during an outage, the whole operation suffers. And if that is truly how a company handles incidents, then it is not a practice I would want to rely on. Good operations teams build processes that protect both the system and the people using it. Communication is one of those processes.
if we are fully down, rest assured there will be no available hands to keep the status page updatedThere is no quicker way for customers to lose trust in your service than it to be down and for them to not know that you're aware and trying to fix it as quickly as possible. One of the things Cloudflare gets right is the frequent public updates when there's a problem.
You should give someone the responsibility for keeping everyone up to date during an incident. It's a good idea to give that task to someone quite junior - they're not much help during the crisis, and they learn a lot about both the tech and communication by managing it.
reply▲GoblinSlayer43 minutes ago
[-] You won't be able to update the status page due to failures anyway.
reply▲This is just business as usual, status pages are 95% for show now. The data center would have to be under water for the status page to say "some users might be experiencing disruptions".
reply▲They just did an update, and it is bad (in the sense that they are not realizing their clients are down?)
> Investigating - Cloudflare is investigating issues with Cloudflare Dashboard and related APIs.
> These issues do not affect the serving of cached files via the Cloudflare CDN or other security features at the Cloudflare Edge.
> Customers using the Dashboard / Cloudflare APIs are impacted as requests might fail and/or errors may be displayed.
> (in the sense that they are not realizing their clients are down?)
Their own website seems down too https://www.cloudflare.com/
--
500 Internal Server Error
cloudflare
>Customers using the Dashboard / Cloudflare APIs are impacted as requests might fail and/or errors may be displayed.
"Might fail"
well it does say that now, so…
which datacenter got flooded?
> In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 05, 2025 - 09:00 UTC
It's a scheduled maintenance, so SLA should not apply right ?
Yeah, their status site reports nothing but then clicking on some of the links on that site bring you the 500 error
reply▲Company internal status pages are always like this. When you don't report problems they don't exist!
reply▲It’s wild how non of the big corporations can make a functional status page
reply▲They could, but accurate reporting is not good for their SLAs
reply▲They were intending to start a maintenance window starting 6 minutes ago, but they were already down by then.
reply▲There is an update:
"Cloudflare Dashboard and Cloudflare API service issues"
Investigating - Cloudflare is investigating issues with Cloudflare Dashboard and related APIs.
Customers using the Dashboard / Cloudflare APIs are impacted as requests might fail and/or errors may be displayed.
Dec 05, 2025 - 08:56 UTC
Interesting, I get a 500 if I try to visit coinbase.com, but my WebSocket connections to advanced-trade-ws.coinbase.com are still live with no issues.
reply▲emakarov44 minutes ago
[-] probably these websockets are not going through cloudflare
reply▲> Investigating - Cloudflare is investigating issues with Cloudflare Dashboard and related APIs.
They seem to now, a few min after your comment
Im much more concerned with customer sites being down which indicates are not impacted. They are.. :/
reply▲> In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 05, 2025 - 07:00 UTC
Something must have gone really wrong.
headmelted58 minutes ago
[-] It's 1AM in San Francisco right now. I don't envy the person having to call Matthew Prince and wake him up for this one. And I feel really bad for the person that forgot a closing brace in whatever config file did this.
reply▲artlovecode51 minutes ago
[-] Agreed, I feel bad for them. But mostly because cloudflare's workflows are so bad that you're seemingly repeatedly set up for really public failures. Like how does this keep happening without leadership's heads rolling. The culture clearly is not fit for their level of criticality
reply▲> The culture clearly is not fit for their level of criticality
I don't think anyone's is.
viraptor42 minutes ago
[-] > I don't envy the person having to call Matthew Prince
They shouldn't need to do that unless they're really disorganised. CEOs are not there for day to day operations.
> And I feel really bad for the person that forgot a closing brace in whatever config file did this.
If a closing brace take your whole infra. down, my guess is that we'll see more of this.
Life hack: Announce bug that brings your entire network down as scheduled maintenance.
reply▲Management is always going to take too long (in an engineer’s opinion) to manually throw the alerts on. They’re pressing people for quick fixes so they can claim their SLAs are intact.
reply▲Yes, the incident report claims this was limited to their client dashboard. It most certainly was not. I have the PagerDuty alerts to prove it...
reply▲They have enough data to at least automate yellow.
reply▲The AI agents can't help out on this time.
reply▲Yes, it’s really ‘weird’ that they refuse to share any details. Completely unlike AWS, for example. As if being open about issues with their own product wouldn’t be in their best interest. /s
reply▲Wow, just plain 500s on customer sites. That's a level of down you don't see that often.
reply▲Yeah that's a hard 500 right? Not even Cloudflare's 500 branded page like last time. What could have caused this, I wonder.
reply▲"A cable!"
"How do you know?"
"I'm holding it!"
I hope it’s not another Result.unwrap().
reply▲singularity200139 minutes ago
[-] maybe this would cause rust to adopt exception handling, and by exception I mean panic
reply▲maxekman52 minutes ago
[-] A precious glimpse of the less seen page renders.
reply▲Unlike the previous outage, my server seems fine, and I can use Cloudflare's tunnel to ssh to the host as well.
reply▲Mine [0] seems to be very high latency but no 500s. But yes, most cloudflare-proxied websites I tried seems to just return 500s.
[0] https://www.merklemap.com/
ransom153854 minutes ago
[-] So. I don't understand the 5 nines they promote. One bad day those nines are gone. So they next year you are pushing 2 nines.
reply▲kingstnap35 minutes ago
[-] Its just fabricated bullshit. It's how all the companies do it. 99.999% over a year is literally 5 minutes. Or under an hour in a decade, that's wildly unrealistic.
Reddit was once down for a full day and that month they reported 99.5% uptime instead of 99.99% as they normally claimed for most months.
There is this amazing combination of nonsense going on to achieve these kinds of numbers:
1. Straight up fraudulent information on status page. Reporting incendents as more minor than any internal monitors would claim.
2. If it's working for at least a few percent of customers it's not down. Degraded is not counted.
3. If any part of anything is working then it's not down. For example with the reddit example even if the site was dead as long as the image server is still at 1% functional with some internal ping the status is good.
its like someone-shut-down-the-power 500s
reply▲AmateurAlert1 hour ago
[-] reply▲reply▲vanyauhalin59 minutes ago
[-] reply▲altmanaltman35 minutes ago
[-] it's like they didn't fully think it through/expect people to actually use it so soon
reply▲ssolarsystem132 minutes ago
[-] downdetectorsdowndetectors didn't detect breakdown of downdetectors with 500 Error
reply▲superdisk37 minutes ago
[-] Lol. The fact that the 4x one actually works and is correctly reporting that the 3x one is down actually makes this a lot funnier to me.
reply▲A wrong downdetectordowntector is worse than a 500 one. :D
reply▲So down²detector was fake all along?
reply▲So DownDetector is down, but DownDetectorDownDetector does not detect it... We probably need one more DownDetector. (no)
reply▲We have one. But according to Down Detector's Down Detector's Down Detector's Down Detector, that's also down.
reply▲Dilettante_54 minutes ago
[-] Well Down Detector's Down Detector isn't down...What we might need is a Down Detector's Down Detector Validator
reply▲This is a fake detector that just has frontend logic for mocking realistic data, you can easily see it in the source code.
reply▲>half the internet is down
>downdetector is down
>downdetector down detector reports everything is fine
software was a mistake
Ehh, so down detector for down detector is up but it is inaccurate.
reply▲great news, schrodingersdetector.com is available!
reply▲At least it's still right in spite of being down.
reply▲I'm just realizing how much we depend on Cloudflare working. Every service I use is unreachable. Even worse than last time. It's almost impossible to do any work atm.
reply▲Rewriting in Rust is paying dividends.
reply▲That's the 30% vibe code they promised us.
Cynicism aside, something seems to be going wrong in our industry.
Going? I think we got there a long time ago. I'm sure we all try our best but our industry doesn't take quality seriously enough. Not compared to every other kind of engineering discipline.
reply▲Always been there. But it seems to be creeping into institutions that previously cared over the past few years, accelerating in the last.
reply▲themafia33 minutes ago
[-] Salaries are flat relative to inflation and profits. I've long felt that some of the hype around "AI" is part of a wage suppression tactic.
reply▲nlitened55 minutes ago
[-] Also “Rewrite it in Rust”.
P.S. it’s a joke, guys, but you have to admit it’s at least partially what’s happening
koakuma-chan49 minutes ago
[-] No, it has nothing to do with Rust.
reply▲But it might have something to do with the "rewrite" part:
> The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they’ve been fixed. There’s nothing wrong with it. It doesn’t acquire bugs just by sitting around on your hard drive.
> Back to that two page function. Yes, I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn’t have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the file is on a floppy disk and the user yanks out the disk in the middle. That LoadLibrary call is ugly but it makes the code work on old versions of Windows 95.
> Each of these bugs took weeks of real-world usage before they were found. The programmer might have spent a couple of days reproducing the bug in the lab and fixing it. If it’s like a lot of bugs, the fix might be one line of code, or it might even be a couple of characters, but a lot of work and time went into those two characters.
> When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work.
From https://www.joelonsoftware.com/2000/04/06/things-you-should-...
windward23 minutes ago
[-] A lot of words for a 'might'. We don't know what caused the downtime.
reply▲The first one had something to do with Rust :-)
reply▲kortilla30 minutes ago
[-] Not really. In C or C++ that could have just been a segfault.
.unwrap() literally means “I’m not going to handle the error branch of this result, please crash”.
mike_hearn25 minutes ago
[-] Indeed, but fortunately there are more languages in the world than Rust and C++. A language that performed decently well and used exceptions systematically (Java, Kotlin, C#) would probably have recovered from a bad data file load.
reply▲koakuma-chan16 minutes ago
[-] There is nothing that prevents you from recovering from a bad data file load in Rust. The programmer who wrote that code chose to crash.
reply▲mike_hearn10 minutes ago
[-] That's exactly my point. There should be no such thing as choosing to crash if you want reliable software. Choosing to crash is idiomatic in Rust but not in managed languages in which exceptions are the standard way to handle errors.
reply▲MegaThorx40 minutes ago
[-] Did you consider to rewrite your joke in rust?
reply▲it's never the technology, it's the implementation
reply▲capnsketch30 minutes ago
[-] If I had a nickel for everytime cloudflare went down.
Then I would have 2 nickels which is not a lot but still wierd that it happened twice.
reply▲cryptonym20 minutes ago
[-] You would have 2 nickels, this week.
It also went down multiple times in the past; not to say that's bad, everyone does from time to time.
TheGilDev18 minutes ago
[-] I’m still glad they’re here to provide great services and help secure the internet for lots of us!
reply▲reassess_blind8 minutes ago
[-] The "half the internet is down, nothing we can do" excuse works great the first time, but doesn't fly the second time in a month.
What solutions are there for Multi DNS/CDN failover that don't rely on a single point of failure?
hasperdi57 minutes ago
[-] Even LinkedIn is now down. Opening linkedin.com gives me a 500 server error and Cloudflare at the bottom. Quite embarassing.
reply▲At least they were available when Front Door was down!
reply▲Claude offline too. 500 errors on the web and the mobile app has been knocked out.
reply▲I had to switch to Gemini for it to help me form a thought so I could type this reply. Its dire.
reply▲OtherShrezzing41 minutes ago
[-] The site is back up, but it feels fairly silly that a platform that has inserted itself as a single point of failure has an architecture that's got single points of failure.
The other companies working at that scale have all sensibly split off into geographical regions & product verticals with redundancy & it's rare that "absolutely all of AWS everywhere is offline". This is two total global outages in as many weeks from Cloudflare, and a third "mostly global outage" the week before.
themafia37 minutes ago
[-] Crop monoculture created the potato famine. We failed to learn the larger lesson. "Hyperscale" is inherently dangerous.
reply▲reneberlin58 minutes ago
[-] I can imagine the horror of pressure of the people responsible for resolution. On that scale of impact it is very hard to keep calm - but still the hive of minds have to cooperate and solve the puzzle while the world is basically halted and ready to blame the company you work for.
reply▲Cloudflare uptime has worsened a lot lately, AI coding has increased exponentially, hmm
reply▲How interesting. As of 00:30 or so I could still access Claude but then it went down with a 500 from Cloudflare and I thought I'd nab a quick something off Slickdeals but that's down too. My own blog is on Cloudflare's `cloudflared` tunnel and it's working just fine, even the cache, so it must be something hitting some specific type of configuration or some shard hitting some region.
And they're back before I finished the comment. Such a pity, I was hoping to hog some more Claude for myself through Claude Code.
ianberdin41 minutes ago
[-] So, I understand correctly that all websites and services want protection from DDoS attacks, and that's basically their number one concern. The second is caching in different parts of the world. So, it's caching and DDoS. But at the same time, nobody wants to use CloudFront from AWS because it’s not that simple yet. And it’s more expensive, while Cloudflare is free. So, what should we do about all this? This won’t do. We’ve created a gigantic bottleneck that controls the entire internet, just like in the movie Mad Max, where he controlled the only source of water. That’s wrong. And we all fell for it like fools. So, the question is, what can be done in this situation? Are there reliable competitors? Are there any fault-tolerant systems for this? The whole problem is that our DNS, and with Cloudflare, they proxy it. So, if their proxy goes down, everything falls apart. What should we do about this?
reply▲chaidhat32 minutes ago
[-] Someone should make an open source system that lets you easily host containers so that if one fails, we can easily switchover across providers. Like Vercel AI SDK but for containers. That is, if docker isnt failing (it is right now cause it depends on Cloudflare)
reply▲drexlspivey37 minutes ago
[-] Who is we? You are free to stop using their service
reply▲sammy225540 minutes ago
[-] Nobody is being forced to use Cloudflare
reply▲ianberdin38 minutes ago
[-] Since everything is absolutely correct, no one forced it; they just provided a good, excellent solution for free, and consequently, the whole internet has gotten hooked on it. As they say, free cocaine causes harm. So, what are the alternatives? What options are there to protect against DDoS attacks and to make a website quickly accessible from different parts of the world? And at the same time, without paying a sky-high price for it.
reply▲Everyone trying to access a site behind Cloudflare is forced.
reply▲Let your hoster take care of the DDoS and stop using the flaky behemoth.
You haven't actually watched Mad Max, have you? I do recommend it.
For us also Digital Ocean, Render, and a few other vendors are down.
At this point picking vendors that don't use Cloudflare in any way becomes the right thing to do.
bigfudge36 minutes ago
[-] Claude was also down (which brought me here)
reply▲I don't want to criticize cloud flare, I love what they do and understand the scale of the challenge, but most people don't and 2 in a month or so like this is going to hit their reputation.
reply▲ianberdin52 minutes ago
[-] I have 10B idea: cloudflare that does not fail so often.
reply▲How about: internet that is actually decentralized.
reply▲ianberdin43 minutes ago
[-] Yes, on one hand, it was so wonderful. Cloudflare came and said, "Yeah, now we'll save everyone from DDoS, everything's perfect, we'll speed up your site," and bam, they became a bottleneck for the entire internet. It's some kind of nightmare. Why didn't several other such popular startups appear, into which more money was invested, and which would allow some failure points to be created? I don't understand this. Or at least Cloudflare itself should have had some backup mechanism, so that in case of failures, something still works, even slowly, or at least they could redirect traffic directly, bypassing their proxies. They just didn't do that at all. Something is definitely wrong.
reply▲viraptor37 minutes ago
[-] > Why didn't several other such popular startups appear
bunny.net
fastly.com
gcore.com
keycdn.com
Cloudfront
Probably some more I forgot now. CF is not the only option and definitely not the best option.
ianberdin13 minutes ago
[-] Thank you for sending these alternatives, they look good. And, of course, the most important thing is that Cloudflare is free, while these alternatives cost money. And they cost hundreds of dollars at my traffic volume of tens of terabytes. Of course, I really don't want to pay. So, as they say, mice wept and jabbed, but they kept gnawing on the cactus.
reply▲It exists and it's called Bunny.net
reply▲Somebody at Cloudflare is stretching that initial investigation time as much as possible to avoid having to update their status to being down and losing that Christmas bonus.
reply▲I just started getting npm errors while developing something; I was like hmm, strange... then I tried to go down to isitdown. That was also down. I was like, oh this must be something local to me (I'm in a remote place visiting my gramps).
Then I go to Hacker News to check. Lo and behold, it's Cloudflare. This is sort of worrying...
This is painful, if I'm not mistaken this is during a scheduled maintenance too ?
Whenever I deploy a new release to my 5 customers, I am pedantic about having a fast rollback.. Maybe I'm not following the apparent industry standard and instead should just wing it.
JeremyJaydan25 minutes ago
[-] I moved away from Cloudflare over a month ago because I didn't understand how they don't have pricing caps for their upgraded plans, they genuinely seem like the mob but I haven't looked any further into it..
Either way it's been interesting to see the bullets I've been dodging.
What service(s) are you using now? What did you move to?
reply▲Perhaps related? My main fiber WAN went out few hrs ago, failing over to Starlink backup. Discovered it’s a cloudflare issue, as my multi-wan setup tests against 1.1.1.1, which suddenly stopped responding (but only from my fiber ISP). Switched to testing 8.8.8.8 to restore.
If it weren’t for recent cloudflare outages, never would have considered this was the problem.
Even until I saw this, I assumed it was an ISP issue, since Starlink still worked using 1.1.1.1. Now I’m thinking it’s a cloudflare routing problem?
ricardo8157 minutes ago
[-] Their uptime over the year is likely faring worse than your average hosting company, DNS provider or CDN.
reply▲cryptonym51 minutes ago
[-] Some may experience more downtime due to their outages than they'd have from DDoS.
reply▲Free accounts seem to be fine, only enterprise accounts seem to be affected.
reply▲phartenfeller1 hour ago
[-] Wow, three times in a month is really crushing their trust.
reply▲8cvor6j844qw_d642 minutes ago
[-] I'll need to checkup on DigitalOcean uptime, may be better than Cloudflare.
reply▲phartenfeller19 minutes ago
[-] My Hetzner servers have been running fine for years. Okay, there were times when I broke something, but at least I was able to fix it quickly and never felt dependent on others.
reply▲LucasLanglois13 minutes ago
[-] Love that Cloudflare put together a participative and community-driven advent calendar!
reply▲chaidhat17 minutes ago
[-] For those saying we have an over-reliance on software -- is there a way to use multiple CDNs for the same frontend website?
reply▲robotfelix48 minutes ago
[-] Our site is fine, including files served by Cloudflare's CDN and Cloudflare Workers, but the Cloudflare dashboard is definitely down.
The Cloudflare status page says that it's the dashboard and Cloudflare APIs that are down. I wonder if the problem is focused on larger sites because they are more dependent on / integrated with Cloudflare APIs. Or perhaps it's only an Enterprise tier feature that's broken.
If it's not everything that is down, I guess things are slightly more resilient than last time?
Isn't it happening a little too often now? Did someone .unwrap in production again?
reply▲nlstitch52 minutes ago
[-] What ever happened to "no deploys on fridays"? haha
reply▲jazzyjackson53 minutes ago
[-] Is it at all achievable to be fronted by a CDN but fallback to the raw server in case the front falls off? Better to be vulnerable to DDoS than be unreachable altogether
reply▲With CloudFlare specifically probably not. IIRC, they require DNS resolution of your domain to operate so if they’re down, I don’t see how you’d change it to route directly to the underlying site.
Even if you could, having two sets of TLS termination is going to be a pain as well.
But then you end up potentially exposing the origin server. This could be an opt-in option though
reply▲nickdothutton18 minutes ago
[-] So many outages now they all begin to swim into 1, what's that 3 or 4 this quarter?
reply▲It's really cool to me that this site is never down with all these outages of major websites.
Representative of having the best developers behind it.
They just don't use Cloudflare.
reply▲How do they handle DDoS?
reply▲yoctosec57 minutes ago
[-] I use Cloudflare because of their Tunnel to protect my Raspberry Pi, but I think I will just use it without the Tunnel now. My main concern is privacy, but I'm not ready to accept so frequent downtime and dependence on them. The whole reason to host self-host was to be independent anyway. Does anyone have a recommendation for that (that is free)? Should I worry about privacy? My name and my city are on the website anyway.
reply▲Checkout tailscale
reply▲yoctosec27 minutes ago
[-] And what about a website I want to make public? I'm just concerned about my IP being visible, like for my personal website or my searxng instance
reply▲Tailscale's control plane uses Cloudflare.
reply▲my tunnels are still working, oddly
reply▲yoctosec29 minutes ago
[-] Now mine works again too, I guess it was a short outage
reply▲Interestingly enough, also some MS/Azure services are down. For example
https://www.office.com/ just returns:
>We are sorry, something went wrong.
>Please try refreshing the page in a few minutes. If the problem persists, please visit status.cloud.microsoft for updates regarding known issues.
The status page of course says nothing
Linkedin -> the same
reply▲For me Linkedin returns the 500 cloudflare error
reply▲At least the 500 error announces ownership.
Imagine how productive we'll be now!
This is getting embarrassing.
reply▲looks like a big one. interestingly, our site, which uses a TON of Cloudflare services[0] — yet not their front-line proxy — is doing fine:
https://magicgarden.gg.
So it seems like it's just the big ol' "throw this big orange reverse proxy in front of your site for better uptime!" is what's broken...
[0] Workers, Durable Objects, KV, R2, etc
reassess_blind47 minutes ago
[-] My sites that use their main proxy are seemingly up and working? Could be a regional PoP issue.
reply▲Moving off of Cloudflare for my personal domain is on my todo list for the holidays...
reply▲reply▲Will it be down for 10 days again? Who knows. Would've stopped using it after the first 10 day outage anyway.
reply▲arunaugustine1 hour ago
[-] They had a scheduled maintenance between 7am and 11am UTC in Chicago. But that should have re-routed traffic not take down internet right?
reply▲I'm in India and we're affected as well.
reply▲Oceania here gang and i think that it is a global issue
reply▲Artur-Defences46 minutes ago
[-] "Scheduled maintenance is currently in progress"
I image the maintenance was conducted like this:
"fix detroit data center bugs, please be very careful, don't mess up like last time :)"
bypass permissions on
reply▲ednevsky47 minutes ago
[-] Notion is also down (haven't seen a comment on that). It's so funny how the biggest companies literally just have their sites not loading because of Cloudflare.
reply▲NKosmatos43 minutes ago
[-] LOL, 500 returned for many big sites…this is going to hurt and make people rethink.
If it’s not DNS, then someone pushed to production on Friday :-)
reply▲dynamite-ready37 minutes ago
[-] Some of the sites I maintain, are fine. But I'm guessing it's just a matter of time?
reply▲Everything i use depend on perfect cloudflare operation workflow,
practically 99% of these services go down.
What magical qualities it has that no competitors form
for its services?
reply▲MarcelGerber43 minutes ago
[-] Just started working for me again (in Germany), both on our own CF-hosted page and on cloudflare.com itself.
reply▲It's configuration error or related to configuration. It always is with this big things.
Nice thing about Cloudflare being down is that almost everything is down at once. Time for peace and quiet.
norskeld34 minutes ago
[-] Damn, I wish CloudFlare being down also affected local development, so I could take a break from doing frontend… :'(
reply▲downdetector is also down
reply▲it being the first google result and serving the exact same error as the pages one is trying to get info from is too funny
reply▲techguy195450 minutes ago
[-] I can still visit some websites that use Cloudflare, but other don't work.
Blender Artists works, but DownDetector and Quillbot dont.
> Monitoring - A fix has been implemented and we are monitoring the results.
> Dec 05, 2025 - 09:12 UTC
MrAureliusR34 minutes ago
[-] Yeah, cloudflare.com is working and the website that first clued me in to the outage (chess.com) is also working.
reply▲virtualritz58 minutes ago
[-] Yeah, and because of this for example Claude Code is down too because the auth goes through CF. F*cking marvelus, the decentralized web ...
reply▲Towaway6948 minutes ago
[-] for me docker is failing with:
unknown: failed to copy: httpReadSeeker: failed open: unexpected status from GET request to https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/....
so coffee time.
CloudFlare: You can't go down if you're never up.
reply▲MildlySerious52 minutes ago
[-] I can't update DNS entries for my domains with Porkbun, because it's "Powered by Cloudflare".
reply▲Heads will roll at cloudfare. E-commerce customers must be furious.
Impossible not to feel bad for whoever is tasked to cleanup the mess.
Especially around christmas. I was about to buy a pair of Birkenstocks. Nope, site is down. Went on to buy a microphone holder, nope, that site is down as well. :) Sure, I'll still get around to it eventually.
reply▲"In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 05, 2025 - 07:00 UTC"
No need. Yikes.
It seems regular reverse proxying and R2 still works, as we use those and seem to be working fine still.
Can't get to the Dashboard though.
digiajay43 minutes ago
[-] Basecamp was down couple of minutes ago and it's back now online.
reply▲I'm looking for cofounders and investors to build a working cloudflare.
reply▲LinkedIn and MEdium are also down as a result
reply▲polaris6448 minutes ago
[-] DownDetector'sDownDetector does not detect that DownDetector's down
reply▲matt321055 minutes ago
[-] Everyone says vibe coding but people are just fine at being incompetent without the AI help
reply▲Sure, but with AI we can automate that incompetence.
reply▲elijah304044850 minutes ago
[-] Pretty awkward. Thought my WIFI was acting up when I wasn't even able to pull up the Cloudfare website to see if something was down. Then, trying to go to Downdetector and that wasn't working either.
reply▲8cvor6j844qw_d648 minutes ago
[-] Interested if its the same issue that brought down Cloudflare previously.
reply▲It's ok to fail. but the most frustrating thing ever is... there's no contact point or supporting team easily and directly accessible.. this is bad..
reply▲chessmaster-hex35 minutes ago
[-] Some big fishes were affected as well... Crunchyroll, Fortnite, LinkedIn
let's wait for the explanation of this one.
reply▲It is up for me.
All the sites that were 500 error before are able to load now.
No engineers from Cloudflare reading hackernews these days? Should update your status page!
reply▲Cloudflare is investigating issues with Cloudflare Dashboard and related APIs.
reply▲Looks like (some) sites behind Cloudflare still work if they do not have caching on.
reply▲It's not simply about caching as we have CDN and reverse proxying which are still running without issue.
reply▲It's ok to fail, but most frustrationg thing is there's no suppoting team or any contact point accessible directly. this is bad..
reply▲They had a few good weeks.
reply▲meindnoch58 minutes ago
[-] Maybe they should stop vibe coding and vibe reviewing their PRs?
reply▲ianberdin53 minutes ago
[-] NPM is down. World is collapsing thanks to Cloudflare.
reply▲One has to wonder how many times or how often proprietary cloud services have to go down before there is a general shift away from using the cloud and "infinite scaling" for everything. For many, many use cases you do not need neither Cloudflare nor Github nor nine nines for everything (which you are clearly not getting anyway). It's obviously not enough with once a year for most businesses, or perhaps once a month. Weekly outages? For how long?
If you host something that actually matters that other people depend upon and, please review your actual needs and if possible stop making yourself _completely_ dependent on giant cloud corporations.
I wonder how many uptime SLAs will be violated this year.
reply▲vimwizard48 minutes ago
[-] seems related to CF tunnels... policies are being enforced but perhaps origin servers are not being properly served.
reply▲neonnbits50 minutes ago
[-] i was watching the climax of "Fullmetal Alchemist: Brotherhood" on crunchyroll and cloudflare went down again
reply▲how long cloudflarestatus.com takes it to detect usually?
reply▲NPM is down as a result.
reply▲Godspeed, Cloudflare, for the fix
reply▲500 internal server error on most things:
500 Internal Server Error
cloudflare
Hashversion34 minutes ago
[-] what's the estimated loss? any guesses or estimations?
reply▲Just in time for the London work day :)
reply▲All those enterprise architects must be fuming now
reply▲alxbenjamin42 minutes ago
[-] It is up again. There will be a lot of hard talk with Cloudflare, I guess
reply▲isaac330750 minutes ago
[-] This is so cool guys. All of us get to lose millions of dollars together so late at night!
reply▲Ooof status 500 someone’s getting fiiiiired!
reply▲My company's services went down as well.
reply▲eval(requestBody).unwrap()
reply▲Hah even Linkedin is showing 500 for me
reply▲tippa12347 minutes ago
[-] Curious to see which big companies were caught flat-footed during the 18 November outage compared with today. In my opinion, if a company was caught out twice, that reflects poor decision-making and urgency. As the saying goes, fool me once, shame on you, fool me twice, shame on me.
If a company was able to overcome all the red tape within three weeks and not be impacted today, that's impressive.
That's quite unfortunate xD
reply▲hax0r133847 minutes ago
[-] This gotta be an attack, no way its configuration error again.
reply▲jonathrg17 minutes ago
[-] Why not? They have been proudly vibe coding for a while.
reply▲odie553359 minutes ago
[-] How is Hacker News still up?
reply▲Because it doesn't use cloudflare duh.?
reply▲From their response headers, it seems like the request is coming from NGINX directly. How do they defend themselves against DOS attacks?
reply▲otherme12349 minutes ago
[-] I have a handful of sites DNS/NS through Cloudflare, with their certificates, and they are working OK.
reply▲grundrausch3n48 minutes ago
[-] I thought they are running classic FreeBSD servers like in ye olde times.
reply▲aw, i cant go on rateyourmusic
reply▲Notion is also down as a result
reply▲sandruso45 minutes ago
[-] it's back on
but wow, it must be stressful to deal with this
neo_tokyo53 minutes ago
[-] Someone's been vibe coding the scheduled maintenance.
reply▲What a joke of a company. They have the internet in the palm of their hands, and yet let vibe coding ambitions ruin their empire.
Time for everyone to drop this company and move on to better solutions (until those better solutions rot from the inside out, just like their predecessor did)
>Go to <social media page> - 500 error from cloudflare
>Google is <social media page> down -> click first link - literally the exact same 500 cloudflare error html from downdetector
I thought we were meant to learn something ... ?
Turnstile seems up still.
reply▲moralestapia1 hour ago
[-] Ooof, this one looks like a big one!
canva.com
chess.com
claude.com
coinbase.com
kraken.com
linkedin.com
medium.com
notion.so
npmjs.com
shopify.com (!)
and many more I won't add bc I don't want to be spammy.
Edit: Just checked all my websites hosted there (~12), they're all ok. Other people with small websites are doing well.
Only huge sites seem to be down. Perhaps they deal with them separately, the premium-tier of Cloudflare clients, ... and those went down, dang.
My small websites are also up. I wonder if they're going to go down soon, or if we're safe.
reply▲otherme12345 minutes ago
[-] readthedocs down is hurting me the most. My small websites are doing OK.
reply▲LinkedIn, Perplexity as well
reply▲Round 2 of Cloudflare outages.
We can now see which companies have failed in their performative systems design interviews.
Looking forward to the post-mortem.
epolanski49 minutes ago
[-] I can absolutely accomplish nothing today...can't download npm packages, cannot login to services.
I've been a Cloudflare fan for the longest time, but the more they grow the more they look like the weak link of the internet. This is the second major outage in less than few weeks. Terrible.
wildcard12101 hour ago
[-] My Shopify store is down. My competitor stores are also down.
reply▲And it's on Friday again — never change, Cloudflare.
Gentle reminder that every affected company brought it upon themselves. Very few companies care about making their system resilient to 3rd party failures. This is just another wake-up call for them.
nekkooo2e50 minutes ago
[-] Perplexity AI shows 500 Internal Server Error
reply▲I was just arguing yesterday to coworkers I would quit tech before helping centralize any more of the internet on Cloudflare as a massive single point of failure.
Thank you, Cloudflare, for again proving my point.
Just experienced this and came here to check, because even their website is down. The referenced link also returns with 500.
reply▲samwreww53 minutes ago
[-] claude.ai is down bc of it :(
good for OpenAI as they're using something else maybe Vercel?
reply▲Update title to “Tell HN: Cloudflare was down”
reply▲LeonenTheDK57 minutes ago
[-] Nice, just got woken up by my outage alarms, just for it to be Cloudflare again. At least it's _my_ problem!
But my goodness, they're really struggling over the last couple weeks... Can't wait to read the next blog post.
>half internet down
>first "is site down" result (downdetector) down
>downdetectorsdowndetector.com: "everything is fine"
>downdetectorsdowndetectorsdowndetector.com: not even responding
>downdetectorsdowndetectorsdowndetectorsdowndetector.com: "everything is broken"
reply▲bytejanitor57 minutes ago
[-] gitlab.com hasn't noticed yet.
reply▲alex_suzuki53 minutes ago
[-] it has now, for me. can't access web UI (SaaS, not self-hosted, obviously)
reply▲I love it, and we wont learn from this again :-) Looking forward for the 3rd outage in a few weeks.
reply▲kUdtiHaEX35 minutes ago
[-] Cloudflare just closed down the incident on their status page without any additional explanation. Sigh.
reply▲andy_ppp38 minutes ago
[-] Just a reminder that every dependency you rely on, both inside your codebase and external services, has a price.
reply▲isaac330749 minutes ago
[-] This is so cool guys!!! We all get to savor this moment and lose millions of DOLL HAIRS together!!
reply▲basisword56 minutes ago
[-] I’m sure everybody learnt their lesson from last months outage and built in redundancy or stopped relying on Cloudflare.
reply▲I wonder if it is another bug , like unwrap, in their rewritten code.
Also, I don't think their every service got affected. I am using their proxy and pages service and both are still up.
Went to ahref to check a domain, saw 500 and came here to check.
I have a few domains on cloudflare and all of them are working with no issues so it might not be a global issue
kinensake51 minutes ago
[-] Every time Cloudflare is down I'm not sure if it's really down or not because most down detector websites use Cloudflare. Lmao
reply▲https://www.cloudflarestatus.com/incidents/hlr9djcf3nyp>We will be performing scheduled maintenance in ORD (Chicago) datacenter
>Traffic might be re-routed from this location, hence there is a possibility of a slight increase in latency during this maintenance window for end-users in the affected region.
Looks like it's not just Chicago that CF brought down...
reply▲yessferatu45 minutes ago
[-] South African here. Down on our side. Huge sites, like our primary news site is down - medical services, emergency service/information etc... all down. It's been like this since 11:00am our time, so about 13minutes now.
reply▲Internet-level companies are having more outages recently. Is the exposed surface area increasing or is the quality of service suffering?
reply▲Interestingly, my site running on workers
https://codeinput.com is still functioning. Worth mentioning that I don't use Cloudflare firewall/caching (directly exposed workers)
yellow_lead47 minutes ago
[-] Is anyone else woken up by this? My company's service is down too. Considering a move away
reply▲Artur-Defences45 minutes ago
[-] "Monitoring - A fix has been implemented and we are monitoring the results."
reply▲seems to have been resolved
reply▲Dilettante_58 minutes ago
[-] "I warned you about Cloudflare bro!!!! I told you dog!"
reply▲dale111048 minutes ago
[-] Tried to watch anime then realized that cloudflare was down...again. smh
reply▲dale111047 minutes ago
[-] Tried to watch anime and then i realized it was down....again. smh
reply▲Who knows, maybe it will be because of C or C++ this time. Or something else.
reply▲Funny how even safe Rust isn’t able to stop vibecoding without a proper validation. And the fact that it's a monopoly isn't so funny anymore.
reply▲dkdbejwi38359 minutes ago
[-] There is no language that makes it impossible to have any kind of bug ever. The safety languages like Rust offer is around memory, not bad configuration or faulty business logic.
reply▲Rust is one of the few languages where I found AI to be very well checked. The type system can enforce so many constraints that you do avoid lots of bugs, and the AI will get caught writing shit code.
Of course, vibe coding will always find a way to make something horribly broken but pretty.
I have noticed LLMs tend to generate very verbose code. What an average human might do in 10 LoC, LLMs will stretch that to 50-60 lines. Sometimes with comments on every line. That can make it hard to see those bugs.
reply▲0xfedcafe50 minutes ago
[-] Yep, that’s what I wrote. It wasn’t a sarcasm
reply