Crazy that Dang literally manages HN in his sleep!

We all knew that but I haven't seen any confirmation before this.

I was personally worried if there was some major outage of the whole world or something the first time hackernews didnt work because I didnt expect hackernews to go down but rather, something even more catastrophic than aws going down must happen (because we see major cloud outage posts)

https://downforeveryoneorjustme.com/hacker-news

This website had many instances of reports, the last I saw were 52 reports in only a short frame of time, the maximum reports on this are 118 it seems.

> In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.

Its okay I suppose, have you figured out who is crawling hackernews so much tho, was it a ddos attack or an AI company trying to get data, doesn't hackernews support an api and I am sure that there are datasets for it too so Its interesting why they might crawl but we all know the reasons why as they have been discussed here.

In a situation like this one, good crisis leadership is essential. dang, HN will help you with tips from vast collected experience (please chip in):

1. Blame: The first thing to do is to point the finger. That doesn't mean analysing the technical issue, which can delay this step and limit your options, but figuring out who is politically easiest to blame. Often, that's the new guy, but outside contractors and vendors without good connections are also a common solution. Even if you are technically responsible for hiring them, you can always push them under the bus with a little skill. This small sacrifice helps unify and motivate the rest of the team.

2. Emotion: Inject your emotion into the situation and make that the implicit, but indisputable priority. Particularly, outrage and anger - This is completely _____. These people are utterly _____ (I'd use all caps, but that's not allowed on HN). Make sure everyone's attention is over their shoulder, on your emotion, and infect the team with it. Threats are an effective tool here - this is a crisis, and anyone who is calm is not emotionally engaged. Otherwise, they won't care enough about this problem - without you driving them, they probably wouldn't care much at all.

3. Speed: Respnsiveness to stakeholders is very important. People need answers now. Give them answers they want to hear, outcomes they will be comfortable with. Don't worry if different groups hear different things. Your team will find a way to make it all work - that's their job.

4. Communication: Good communication is essential. Make sure you clearly tell your team what they should be doing; repeat it several times to prevent misunderstanding. Especially people with experience can have minds of their own; keep them on track. The situation is a crisis so you can't take any risks; stay on top of them and everything they do, and give input if you're not certain they are doing what you would be doing.

5. Victimhood: Find a way to turn the tables: Make it about you, and how you're the victim here, and feed the fire with more outrage. With this and outrage, nobody will undermine the team by challenging your ideas or authority, which is the most essential component of a successful outcome.

Have I missed anything?

Maybe it would be fine if ops alerts were silenced during normal US sleeping hours?

HN is important, but unlikely much harm could be done before morning.

(Source: Lost a lot of sleep at one place, enough to realize that sleep interruption and deficit has significant costs.)

No apology needed. We all needed to stop procrastinating anyways :)

I’d love to know more about what running a site like HN involves, would be great to get a write up of what it’s like running something like this at this scale (and what kind of traffic you guys get)!

Frankly, I don't understand why someone would even try to crawl Hacker News.

There is an official dump which doesn't even require parsing HTML at all: https://console.cloud.google.com/marketplace/details/y-combi...

The transparency is deeply appreciated by me and others. We don't pay to keep HN on, so we cannot complain. Thank you and the rest of the team for all you do to give us a corner of the internet that is quite 'different' from the rest of the wild west that is the web.

Short lived and driven by good intentions– all's good. Thanks again for keeping this thing going!

During the last week my IP was banned for unknown reason. Glad to hear it might not be a problem from my side.

Can't speak for others, but I'm sure i'll be pretty fine if no one gets woken up if HN is down...

Of course, they'd better restore service after they wake up naturally, because I need my HN dose. But it's not worth losing sleep over it.

> The reason I did that is that the anti-crawler protections also unfortunately hit some legit users

How does this happen?

I didn't realize you were carrying the pager too! Kudos!

Even after providing firebase endpoint, crawlers come to the site ?

dang!

Decades ago I had to write a Perl script to auth to the site for proper downtime checking. Some things never change :) Good luck with the triage.

Looking forward to the post mortem. :)

Sometimes I'll catch myself absentmindedly reopening the browser and checking two or three front pages, seconds after having just checked them and closed the browser.

Yeah me too. Wake up -> HN down -> That's weird, oh well it's usually only down for a few minutes -> I should check if HN is still down -> That's weird, oh well it's usually only down for a few minutes -> I should check if HN is still down -> loop.

That was a few hours ago. I'm glad this loop is broken.

Yeah, I had assumed something very major must have happened for HN to go down, lmao and I even asked in some linux discord server regarding it asking if there is a major outage as hackernews is down

I woke up and was wondering if I’ve just woken up in hell!

HN is how I discover whether other sites are down or not, so it serves a critical function, so of course I check it frequently.

/s

I learn more reading the comments here than anywhere else. Thanks everyone for my addiction.

I'm glad I'm not the only one. If I type 'n' into any browser it autocompletes to HN.

If you're looking to put the brakes on that, I've used LeechBlock to add a 5-second timer to opening a new HN window (along with other block schedules). The timer even fails if it loses focus, so it really helps slow you down.

https://www.proginosko.com/leechblock/

    echo '127.0.0.1 news.ycombinator.com' | sudo tee -a /etc/hosts

I say. Vibe coded 4 apps once I got past that, on my way to half a billion in ARR already.

I’ve turned on no procrast mode and set it to ten minutes per hour. Helped me a lot!

Same! Right there with "every day must begin with coffee"

It just reinforces for me that addiction is a human problem not a problem with technology

I know dang basically works tirelessly to not change the format in order to not induce those addictive patterns

but yet here we all are

⌘-T, N, <RET>

Did it like 5 times during that 1h-ish outage. :(

So do I, but it was such a shock that I just passed out, and when I woke up, it was back up.

What? You mean you ... close the HN tab?

I did not know how addicted I was to HN until today lol

I already knew that. :)

obligatory: https://xkcd.com/477/

You should add a graph of visitors per-minute for the status page for the past 24 hours or so. Would really help for situations like this

Thank you. I was thinking myself or my corporate IP was shadowbanned

Is this a mistake by hund, or the configuration of hund by HN?

And with this, you should shut down hn.hund.io because you just did massive damage to your brand's value. If it is unofficial, if you can't do it right, don't do it at all. You had one job, the thing your company exists to do, and you blew it. Nothing makes me rage more than a lying dashboard, and yours lied for 3+ hours.

This was more impactful to my day than the last AWS and CloudFlare outages...

https://xcancel.com/HNStatus only uses chronological ordering (after any pinned tweets) and that has the last message 12 Dec 2023.

(Basically whenever you see an x.com link just change it to xcancel.com and avoid the nonsense.)

Only way I have figured out how to to change the "Following" sort order back to chronological is from the mobile app: click the down arrow on the "Following" tab. Change the sort from "popular" to "most recent."

Seems to reset it on the web view, too.

https://hn.hund.io/ Is a status page, no idea if official or not, but it didn't register here for some reason.

I didn't read the post text, it's identified there haha, my bad! I wish the text post text wasn't grey, I gloss over it too easily.

I always hated the late use-it-or-loose-it at the end of the year where you end up buying the things that were denied requests from earlier in the year. You just cost me half a year of using the damn thing.

> news.ycombinator.com/status page

That's not so useful when news.ycombinator.com is having problems.

Next time you can avoid that fate by opening HN in a private browsing (or whatever your browser calls its equivalent) window. This outage, like the vast majority of HN outages, only affected logged in requests.

I suppose you could also just clear your HN cookies in regular browsing window, but then when they fix it you'd have to log in again.

hahaha

A lot of them got fooled by the caching; pages for signed-out users are cached heavily and those kept returning successful responses even if the actual backend server was down.

And an outage reliability indicator for that outage reliability indicator.

So a downdetectorsdowndetector.com but for Hacker News?

Interestingly it stayed up if you weren't logged in.

Next time try not to beat a dead horse expecting it to resurrect. :)

Now it makes sense. I was puzzled about why it was working on the phone browser and not on my system. I'm logged into HN on my system.

You don't visit enough my friend :)

It's down about 8.4 minutes per week. On 26% of days it doesn't work at least once, and on 12% of days it has more than one consecutive failed check. The longest uptime streak was 24 days

I've been keeping track since exactly 2 years (to the day!) because I was surprised that it seemed briefly down for me on a daily basis. Was I getting unlucky and hitting it every time, or was it just down very often? Nobody posted anything so I started answering the question for myself :p

I've been meaning to post the tracker to HN but there's a pesky bug I want to fix: the "is it currently down" stat. I don't know how this is beyond me but something in the code bugs out. So this is my first time posting about it

Time to update the meme: https://preview.redd.it/cloudflare-had-a-rough-day-today-and...

We can live without the former, but not HN!

you could refuse to be angry to mock it

You handle your irrational anger much better than whatevermrfukz who keeps pooping his pants.

I thought I was being rate-limited for opening posts too fast, which has happened before.

After more than an hour I thought, "wow this is pretty harsh" and "so much of my exposure to learning things is directly tied to HN posts". I was lost lol.

Me too and I was wondering what I did!

Your comment history doesn't sound IP-ban-worthy. Is there an alt that has you worried? :P

Thank you for HACK. I love it!

You use ChatGPT to know if a site is down?

Keep in mind that votes aren't supposed to be about whether you agree or disagree, but whether a comment adds substance to the thread. A good question that makes a wrong assumption can be worth upvoting; a correct but fairly irrelevant statement about politics not (to give a relatively obvious example)

Being "voted to -2" doesn't necessarily mean you were wrong (it often correlates though). People might just think it wasn't relevant in whatever context you posted it in

I often find it hard to tell what makes people think something I write is not helpful (or sometimes also a comment someone else made) and thus appreciate comments that clarify constructively. It can also help to ask for clarification if you're particularly surprised about the votes on a given post

Unless the downtime was caused by something Cloudflare would've prevented, this downtime would've happened regardless of being behind Cloudflare. Cloudflare adds another single point of failure.

Could this be a self-inflicted bug? In that case, the broader point still stands: cloud providers can cause outages that are outside your direct realm of responsibility.

Complaining about downvotes usually gets you more downvotes, so the response here isn't really that surprising.

You're probably being downvoted today for your "I was smarter than you" tone.

It's absolutely irresistible downvoting people who preemptively complain about being downvoted like you do. It really made my day. Post another complaint so I can do it again please! It's not knee jerk when you explicitly ask for it, by leading with a complaint about downvoting, instead of just making your point and letting it fall or rise on its own merits. You're the one who put the idea of downvoting you into my head in the first place.