I think often about Mahalo, the sleazy shovel content that was spamming the web back in 2007. Google shut that down somewhat fast, although it did take several years. These days with AI and more aggressive spammers it's a losing battle. The real problem is the financial incentives that make this kind of spamming profitable in the first place.
My tiny little blog gets about 3 requests a week for someone to "pay me to run a guest article". Going rate is $50-$200 and again, my blog is tiny.
https://housefresh.com/how-google-decimated-housefresh/
On HN here:
https://news.ycombinator.com/item?id=40239811
And their previous article mentioned in that post generated a lot of discussion on HN:
Sometimes dumb, bruteforce and biased solution can work way better than any automation you can come up with
My best guess is it's because they finally have a real competitor in ChatGPT.
> The real problem is the financial incentives that make this kind of spamming profitable in the first place.
Yeah, but the financial incentives exist on both ends. There's a gross symbiotic relationship between Google and SEO spammers, because Google also owns the ad network the spammers put on their page. If Google puts ad-laden SEO blogspam as the top result and a user clicks it, the user sees a bunch of ads from Google. Everyone wins: Google, the SEO spammers, and advertisers. Well, everyone except the user, but who cares about them?
My guess/hope is that ChatGPT has made someone who actually cares about the quality of search results actually step in and say things have gone too far.
Wasn't there a big story last year in the wake of the DOJ antitrust investigation about Google manipulating search quality to boost ad revenue? I can't put my hands on a reference now, in part because Google is so bad at search these days I can't find anything more than a few months old.
Ironically, I probably would have paid the same amount to Google for ad-free, old-style (accurate) Google searches, but no, they just wanted to keep cranking that ad dial up every year so that ship has sailed.
At this point, I'm enjoying watching the old guard of search scrambling to find a life jacket.
Google is getting destroyed by the chatbot workflow because it is no longer the start of a browser session and clickthrus (the things that earn the high sponsored link rates) are falling as more users get their queries answered faster with less effort.
Define "wins". From what happening right now, it seems that google may lose much more than it earned by aligning with seo spammers
Maybe they need to start locking employee stock options for 100 years to prevent them optimizing short-term gains?
It works great, until it doesn't. But that's a problem for the next CEO.
Meanwhile $company's page was company.tld/234897234982-029823749823742-2340823492 and 3 pages down was a phone number if your browser didn't choke on the javascript.
For ISP ones, I recommended people print a copy so they can call if they can't get on the internet, which kinda backfired when a major ISP changed their tech support number (!) and it redirected to a toll-free squatter's sex chat line.
Turns out $company really hates it when you call their call (cost) centres.
I had maybe 50 pages for our different oligopolies and averaged $500/month revenue on adsense, so GOOG's cut was $250/month.
Today, for one $company, the first 9 results are different pages from $company.tld, each unhelpful with a phone number in their own way, and they don't run adsense!
My guess is it's because a bunch of articles about it have been posted to hn recently.
Bingo. I always chuckle when people here say Google has lost it, and become incompetent. Well, they all make the mistake of assuming that they’re trying but failing, rather than that it’s deliberate simply due to boring economics.
Now look at how quickly decades-long problems, so big they have an entire cottage industry built around it, suddenly be cleaned up. Incompetence? Nah.
Of course, this does nothing to convince regulators and not even average HN user that innovation is harmed by these dominant players. Someone’s gotta think of the poor mega-corps.
What would it take for someone to make it today? No AI, only 1 on mobile, and sites with heavy ad loads are punished
I also like my "Before AI" lens I can click on to search the internet pre-2021. And you can downrank or fully block those garbage spam sites. They even have a "leaderboard" for most blocked/pinned sites you can use to get started.
The entire thing was so blatant and obvious that I assumed Google did not care due to ad revenue.
When ChatGPT launched search, you could immediately skip over all the crap. It made search nice again.
Yeah but that was before they hired the incompetent grifter Prabhakar Raghavan and eventually made him head of Search.
But people were complaining about the sAme issues under Matt Cutts. Also, there has been A Ton more money and work chasing the SEO farm game. Now big private equity companies have focused on buying a stable of big brands to do the same that used to be garage startups.
It would, as this scam relies on the domain authority, regardless of their "offline authority".
What you really want is the internet from 5-10 years ago (really even longer than that), and that's not coming back.
The internet or rather the crawlable WWW itself has changed.
The WWW content itself is no longer what it was.
seems like local news sites are the absolute worst in this area
I feel like testosterone therapy should be more readily available….
Web search has always been an extremely messy solution to many problems. Think about the premise: type in anything, and somehow it will read your mind, intuit who you are and what you really wanted, find the exact thing amid the morass of the whole web, and then give it to you?
That's impossible. So it uses tricks to make it seem like it worked. It uses information about you to refine results. It uses curated, human-edited search and result heuristics for the most common or difficult search queries. It uses a giant corups of data, and shows you things that are like what you wanted.
You don't notice that it isn't giving you the best result, because there are so many mediocre-but-acceptable results to look at. And it doesn't have to work perfectly every time, because we can "sift through" results and "refine" our search. Often we are flooded with results that are targeted at us, rather than what we want, because, remember: Google is an advertising company, and the entire Web is now a shopping mall, where either you're being sold-to, or you're just being sold.
You will get results, and they will sort-of seem like what you wanted, so you will just sort of sigh and accept it. Because what other option is there?
There are more intelligent, more accurate, more safe, ways to solve the problems people have, that are not "a search engine". It's time we start implementing them.
> type in anything, and somehow it will read your mind
I think we can go back to the way things were, which had nothing to do with mind reading. In the past, you could type in word, and google would offer 10 million results, and you could page through each of them. That was very powerful, and google does not do that today.
That's fine. It's always been fine. I don't need Google to read my mind and fulfill my dreams.
The problem isn't that they're not divinely perfect. The problem is that they used to be good enough, and now they're not.
> There are more intelligent, more accurate, more safe, ways to solve the problems people have, that are not "a search engine". It's time we start implementing them.
What solutions are there that fulfill all the use cases of a search engine, while definitively not being a search engine? An AI chatbot that gives me synopses of the same websites that I was searching for does not count.
I never once asked for anything remotely like this. Maybe you could just show me results for the fucking thing I typed? When I go to the library, the Dewey decimal system doesn't rearrange itself based on all the metadata the library has on me and people fitting my demographic criteria, it just shows me what I fucking searched for.
Ironically, Google itself was a key developer of that tech.
If there is any solution it would seem to involve removing the incentive to merely look at your page. That problem seems remarkably stubborn.
I think this is the root cause of the problem. Google can easily put a big dent in this problem by allowing users to create their own importable/exportable filters and support the dissemination of something like "EasyList for search results." But that kills their golden goose of advertising influence.
Who will be in charge of curating that list? We know that crowd-sourced stuff is easily abused (see Amazon reviews, see YouTube comments).
And unlike Amazon reviews or YouTube comments, anyone can fork it if they think they can maintain it better.
[1] "The filter lists are currently maintained by four authors, Fanboy, MonztA, Khrin, Yuki2718 and PiQuark6046, who are ably assisted by an ample forum community." https://easylist.to/
I'll give you a concrete example of that and it is a right old pain.
Let's try upgrading Debian Bullseye to Bookworm. Search "upgrade debian bullseye to bookworm" - first hit from DDG is: https://www.debian.org/releases/bookworm/amd64/release-notes... - YES - Debian documentation, staid, verbose, stolid and correct.
Now let's try to upgrade a Raspberry Pi from Bullseye to Bookworm: Search "upgrade raspberry pi bullseye to bookworm". First hit: https://raspberrytips.com/upgrade-raspberry-pi-os-bookworm/.
There are loads more hits like the above and they are nearly all wrong. The RPI distribution is based on Debian Linux but has a few differences. Between those two versions of Debian, RPi changed things in /boot quite dramatically and failing to do that, you will end up with a weird chimera - I created several of these beasts until I fixed them: https://blog.scheib.me/2024/04/14/upgrade-raspberry-bullseye...
In this case it may actually be a blog matching the template of the AI clones! However, they do all look very similar.
DDG has never been very good.
Given Kagi's abysmal adoption rates, it's clear that good search isn't worth it for most people.
These days my default assumption is that any SAAS product will get worse and more expensive over time, so it has to be pretty good to justify reworking my online habits around, given that I don't know how long I'll keep using it. Hopefully Kagi will be the exception to that rule, but I wouldn't bet on it.
But that’s also the answer on preference. Google is good enough for most people. For everyone else, there can be a paid premium layer. Similar to news, this might be the equilibrium, not an anomaly.
1. Make HTTP
2. Send HTTP via TCP
3. Perform text processing on the response body (I create own SERP instead of using Google's)
Personally, I use multiple programs, some I wrote myself in C, to perform these individual steps, connected by UNIX pipes and the shortest, simplest possible Bourne shell scripting
However there are countless ways to perform these steps in wide variety of programming languages; there is no need for UNIX or shell scripting, it is purely personal preference
Details here: https://larslofgren.com/cnn-usa-today-forbes-marketplace/
A lot of people have a significant financial incentive to win at that search query.
What would the perfect top search result for that look like?
It would probably be an article written by professional writers in a trustworthy publication with a strong ethics policy, provably followed over the years, concerning whether they accept payment for promoting specific products in supposedly impartial reviews.
If you can figure out how to algorithmically detect that kind of content you could build a pretty great search engine!
And people do transfer trust from the medium to the product, so neither is the second filter very robust when it comes to people perception
How is Forbes worse than any of those shallow comparison pages?
* cornbreadhemp.com * forbes.com * healthline.com
Forbes did make it on the blocked and lowered leaderboards.
forbes.com healthline.com medicalnewstoday.com fortune.com
But I usually use site:reddit.com in my searches anyway so it doesn't matter to my personal habits.
Now let's make corporate stock manipulation illegal again and ban corporate stock buybacks. Talk about a purely manipulative business strategy.
https://hbr.org/2020/01/why-stock-buybacks-are-dangerous-for...
https://www.theatlantic.com/ideas/archive/2018/07/are-stock-...
-Stock buybacks are not manipulation, they’re simply a way to return cash to shareholders and then the shareholder decides when to incur tax liability. A company is well within its rights to issue additional shares or buy back and destroy shares at their discretion. It’s functionally equivalent to a dividend without a taxable event.
-Corporate boards award stock grants to executives because they want management to be aligned with shareholders. I think executive compensation is excessive, but stock grants do align management and shareholders.
If your search for some specific term "$foo", nearly every result is just 'search site $bar for "$foo"', taking you to the site's search page, regardless of whether $foo is actually found on the site.
The search behavior is the same whether $foo is a popular generic term or something niche.
Seems like a referendum for a name change would be a good idea, considering the current typical usage of that term.
> Consumer reviews from trusted websites like Healthline, CNET, and Forbes Health that provide in-depth reviews and rankings based on effectiveness, ingredients, and customer feedback.
So the LLMs are now giving us affiliate link garbage, but we can’t easily see that was the source, and the affiliate links don’t even work. Everyone loses in this scenario.
They could even scale the downranking so that the higher your site's reputation, the more it gets downranked if you're advertising poorly-reviewed companies. That would ding Forbes more than it would ding Joe's Little Blog, and prevent highly ranking sites (like Forbes) from having a monopoly on some search results.
I remember this guy
pointing out the line between what you can get away with with SEO and what you can't get away with and what you can't get away with is making Google look stupid.
Yes, this is /blog. The whole blog.
That page gives a good hint at how opaque these paid placements can be to an outsider like Google. Really tough to prevent too much collateral damage when going after bad actors like Forbes. Glad Google is working on it though.
When I search for “Best Delta credit card”, CNBC shows up 6th