Recently I found myself manually adding `/llm.txt` to most websites I visit because I find the content for LLMs strait to the point and clear. The only annoyance is web browsers like chrome do not render the markdown.
So could the AI revolution actually fix the web for humans as a side effect?
Do you find yourself doing the same?
It's the law of monetization.
And despite this, modern life is made possible by the illusion that "regulations" work..
If I remember correctly this "standard" was setup by someone but without involvement of any of the major AI players.
All regular pages are aggressively scraped to the extent it's a problem I have to consistently manage, but not llms.txt.
But perhaps these are developers specifically targeting these pages to feed whatever LLM they are using.
But nah, I'm sure OP doesn't know about CDNs.
So it get even stranger, I am the only one reading those /llms.txt ...
https://developers.openai.com/llms.txt
https://docs.anthropic.com/llms.txt
https://geminicli.com/llms.txt
https://github.com/llms.txt
https://docs.aws.amazon.com/llms.txt
https://openrouter.ai/docs/llms.txtanyone who's, even slightly, clued into how agents access documentation, has been making changes to their pages. ex: https://searchtxt-web.fly.dev/search?q=aws
https://github.com/AnswerDotAI/llms-txt/issues/2
It’s been completely ignored ever since.
https://news.ycombinator.com/item?id=48411569
BTW why should Chrome even consider rendering a .txt file as markdown?
There is an enshittification cycle at work. The web used to be good, predominately text, and useful, 25 years ago. Then... slowly... we added javascript, then AJAX, CSS, flash, interstitials, popups, marketing, social media, algorithms, doomscrolling... gradually but surely turn it into the unusable cesspool that it is today.
Now we have AI! I think a big part of its utility is that it gets us back to text/information, and lets us bypass all the "beautiful" design / nonsense on the material it is trained on.
However, AI is just beginning its enshittification cycle - now that it has a critical mass of users, it is an irresistible target to start slowly adding ads, misinformation, conspiracy theories, and whatever else people can dream up, until it also becomes unusable and the cycle repeats.
Result: no such item.
From where do you got the idea that adding /llm.txt to urls will produce markdown?
This is just a redeux of the early web.
I imagine Claude could zero-shot a Chrome plugin for that.