FilterHN

Codex scraped the ICM website and discovered 2026 Fields Medal winner list

46 points

by zaikunzhang

1 hour ago

| past

| 9 comments

| phemex.com

| HN

▲

edoceo

36 minutes ago

[-]

I've been working on a site. It's new, domain is only a few weeks old. It's got SSL, so all the bots know it exists. It's never had any sub-pages exposed, just the placeholder lander, no links.

Somehow in Google search one of the unguessable pages is indexed. We have used Claude and Gemini to assist with some design aspects.

I'm thinking some aggressive data ingestion/indexing is happening by all the bots in the quest for frontier models.

▲

cynerx

9 minutes ago

[-]

Are you using Cloudflare by any chance? I think the Crawler Hints setting [1] exposed some of my "secret" pages in the past.

[1] https://developers.cloudflare.com/cache/advanced-configurati...

▲

resonious

29 minutes ago

[-]

I've also seen Google indexing pages with random values in the path that don't get linked to statically (server asks for the URL then redirects to it immediately). I'm pretty sure they index straight out of the Chrome address bar.

▲

st_goliath

15 minutes ago

[-]

Yep. I remember a similar story as GP described from a friend back in 2008. The site he was working on that wasn't linked to yet was suddenly indexed after he checked out what it looked like in the fancy new "Chrome" browser that Google had just released, causing some moderate panic on his end.

▲

foobarbecue

24 minutes ago

[-]

Holy crap I hope that's not true. I've also had unguessable pages indexed, though, and don't have an explanation.

▲

nicce

18 minutes ago

[-]

Something worth inspecting further. We know that Chrome stores and sends the browsing history but this is an interesting vector.

▲

DANmode

4 minutes ago

[-]

I’d be more surprised if they weren’t capturing this information.

Especially if you have autocomplete-while-searching type of features on.

▲

pohuing

4 minutes ago

[-]

There's a couple avenues besides just stealing what's in your URL bar.

If you don't use wildcard certs all of your subdomains can be scraped from the certificate transparency logs. Additionally, any domain+cert using HSTS with preload enabled end up in a big list at Google to speed up the initial connection from browser to site.

▲

brookst

28 seconds ago

[-]

For hosts, but not pages on the site.

But I think the other explanations take care of pages: cloudflare hints, chrome reporting addresses visited, etc.

▲

fragmede

1 minute ago

[-]

CT logs just explain how they found the domain. T doesn't explain how they could have found unlinked content on the domain itself. If I put up secret-example.com/asdf-1234567.html, how does that page get found if there are no public links to it?

▲

dreambigwrkhard

14 minutes ago

[-]

Depending on the CMS, if it's wordpress (15% chance, ha) there is a sitemap function built-in out of the box. The bots don't need to guess.

▲

ashu1461

53 minutes ago

[-]

Someone used Codex to scrape the ICM website schedule and discovered that the winners list was simply hidden in the front-end code with a "hidden" tag

This is on the devs and feels like a very basic leak which could have exploited in the non LLM world as well.

▲

st_goliath

13 minutes ago

[-]

Well, the angle is kind of important here. The company gets their name in the news, they have a reasonable explanation why they were scraping around, and we end up with a story about innovative tech company whiz-kids who made a funny discovery, while it was the webdevs on the other side that goofed up.

Imagine a private individual just scraped the website (or simply clicked 'view source') for no reason in particular and then told people about it... They'd be labeled an uber-haxxor, face a civil lawsuit asking for ridiculous damages while being threatened with a prison sentence over CFAA violations. Hell, that might even drive some people to suicide.

▲

ajb

33 minutes ago

[-]

Yeah that happens all the time. Anyone/thing with popular public releases has fans/journeys scraping the website looking for unreleased material or scoops.

In the early days one of the high profile soaps in the UK published their "catch up" summaries for the week ahead which you could get just by editing the date in the URL. But back then not so many people were looking, so they were doing it for months...

▲

efficax

16 minutes ago

[-]

twist: codex also wrote the code that placed the winners list in a hidden element

▲

zaikunzhang

38 minutes ago

[-]

Related to the earlier discussion:

https://news.ycombinator.com/item?id=48902814