Google removes AI health summaries after investigation finds dangerous flaws
97 points
3 hours ago
| 14 comments
| arstechnica.com
| HN
myhf
2 hours ago
[-]
If an app makes a diagnosis or a recommendation based on health data, that's Software as a Medical Device (SaMD) and it opens up a world of liability.

https://www.fda.gov/medical-devices/digital-health-center-ex...

reply
InMice
2 hours ago
[-]
Not surprised. Another example is minecraft related queries. Im searching with the intention of eventually going to a certain wiki page at minecraft.wiki, but started to just read the summaries instead. It will combine fan forums discussing desired features/ideas with the actual game bible at minecraft.wiki - so it mixes one source of truth with one source of fantasy. Results in ridiculous inaccurate summaries.
reply
yakattak
1 hour ago
[-]
I find its tricky with games, especially ones as updated as frequently as Minecraft over the years. I've had some of this trouble with OSRS. It brings in old info, or info from a League/Event that isn't relevant. Easier to just go to the insanely curated wiki.
reply
Loughla
1 hour ago
[-]
I run a small business that buys from one of two suppliers of the items we need. The supplier has a TRASH website search feature. It's quicker to Google it.

Now that AI summaries exist, I have to scroll past half a page of result and nonsense about a Turkish oil company before I find the item I'm looking for.

I hate it. It's such a minor inconvenience, but it's just so annoying. Like a sore tooth.

reply
lelandfe
7 minutes ago
[-]
You can cure Google's AI (but not bad results) with &udm=14

https://google.com/search?q=parkas&udm=14

reply
hsuduebc2
1 minute ago
[-]
They AI summary is total garbage. Probably most broken feature I saw being released in a while.
reply
mannykannot
7 minutes ago
[-]
The fact that it reached this point is further evidence that if the AI apocalypse is a possibility, common sense will not save us.
reply
dreadsword
2 hours ago
[-]
"Dangerous and Alarming" - it tough; healthcare is needs disruption but unlike many places to target for disruption, the risk is life and death. It strikes me that healthcare is a space to focus on human in the loop applications and massively increasing the productivity of humans, before replacing them... https://deadstack.net/cluster/google-removes-ai-overviews-fo...
reply
bandrami
2 hours ago
[-]
Why does healthcare "need disruption"?
reply
seanmcdirmid
24 minutes ago
[-]
Many people don’t get it, it’s really expensive, even in countries with non broken healthcare systems (not the us) costs increase rapidly and no one is sure how the systems will remain solvent with the same level of care given today. The way things are currently done are entrenched but not sustainable, that’s when disruptions are apt to appear.
reply
wswin
2 hours ago
[-]
It's inefficient and not living to its potential
reply
bandrami
2 hours ago
[-]
And "disruption" (a pretty ill-defined term) is the solution to that?
reply
__loam
1 hour ago
[-]
The solution is single payer. Any attempt to solve this with technological band aids is completely futile. We know what the solution is because we see it work in every other developed nation. We don't have it because a class of billionaire doners doesn't want to pay into the system that allowed them to become fabulously wealthy. People who are claiming AI is the solution to healthcare access and affordability are delusional or lying to you.
reply
reaperducer
22 minutes ago
[-]
It's inefficient and not living to its potential

Yeah, because we saw what a great job the tech bros did making government more efficient.

reply
plagiarist
1 hour ago
[-]
The inefficiency is the buying of yachts for billionaires.
reply
eru
1 hour ago
[-]
Compare: Google's founders can buy all the yachts they could possibly eat, yet Google Searches are offered for free.

If we could get healthcare to that level, it would be great.

For a less extreme example: Wal-Mart and Amazon have made plenty of people very rich, and they charge customers for their goods; but their entrance into the markets have arguable brought down prices.

reply
__loam
59 minutes ago
[-]
And Google search, a service on the level of a public utility, has been degrading noticeably for years in the face of shareholders demanding more and more returns.
reply
harpratap
1 hour ago
[-]
Because insurance companies incentivize upward price momentum. The ones who innovate and bring the prices down are not rewarded for their efforts. Health inflation is higher than headline inflation because of this absence of price pressure
reply
eru
1 hour ago
[-]
I sympthatise with the argument. We should test it against real world data.

Eg your argument would predict that healthcare price inflation is not as bad in areas with less insurance coverage. Eg for dental work (which is less often covered as far as I can tell), for (vanity) plastic surgery, or we can even check healthcare price inflation for vet care for pets.

reply
threethirtytwo
39 minutes ago
[-]
This argument doesn’t make sense to me. Insurance companies are structurally incentivized to minimize payouts across the board. They want hospital bills lower, physician compensation lower, and patient payouts as small as possible. If insurers had unilateral power, total medical spending would collapse, not explode.

The real source of high medical costs is the entity that sets the hospital bill in the first place.

The explanation is much simpler than people want to admit, but emotionally uncomfortable: doctors and hospitals are paid more than the free market would otherwise justify. We hesitate to say this because they save lives, and we instinctively conflate moral worth with economic compensation. But markets don’t work that way.

Economics does not reward people based on what they “deserve.” It rewards scarcity. And physician labor is artificially scarce.

The supply of doctors is deliberately constrained. We are not operating in a free market here. Entry into the profession is made far more restrictive than is strictly necessary, not purely for safety, but to protect incumbents. This is classic supply-side restriction behavior, bordering on cartel dynamics.

See, for example: https://petrieflom.law.harvard.edu/2022/03/15/ama-scope-of-p...

We see similar behavior in law, but medicine is more insidious. Because medical practice genuinely requires guardrails to prevent harm and quackery, credentialing is non-negotiable. That necessity makes it uniquely easy to smuggle in protectionism under the banner of “safety.”

The result is predictable: restricted supply, elevated wages, and persistently high medical costs. The problem isn’t mysterious, and it isn’t insurance companies. It’s a supply bottleneck created and defended by the profession itself.

Insurance companies aren't innocent angels in this whole scenario either. When the hospital bill fucks them over they don't even blink twice when they turn around and fuck over the patient to bail themselves out. But make no mistake, insurance is the side effect, the profession itself is the core problem.

reply
sergiotapia
1 hour ago
[-]
because in america at least, the supply of doctors is kept artificially low. that combined with exploding administrative headcount, means patients are getting pretty terrible, expensive service.
reply
dyauspitr
1 hour ago
[-]
Seriously? Spending a night in a hospital results in a $10,000 bill (though the real out of pocket is significantly cheaper. God help you if you have no insurance though). Healthcare in the US is the thing that needs the biggest disruption.
reply
gmueckl
1 hour ago
[-]
But no business is going to fix it. The market is captured. Only a radical change of insurance laws is going to have any impact. Mandate that insurance must be not for profit. Mandate at least decent minimal coverage standards and large insurance pools that must span age groups and risk groups.
reply
eru
1 hour ago
[-]
Many hospitals are already non-profit. That doesn't seem to bring down prices. Why would you think that this would work for insurance?

Profit isn't even a big part of the overall revenue.

> Mandate at least decent minimal coverage standards

I assume you want higher coverage standards than what currently exists? Independently of whether that would be the morally right thing to do (or not), it would definitely increase prices.

> and large insurance pools that must span age groups and risk groups.

Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).

reply
kunai
1 hour ago
[-]
These solutions are often proposed as easy fixes but I'm skeptical that they actually will do much to reduce healthcare costs. Healthcare is fundamentally expensive. Not-for-profit hospitals and for-profit hospitals don't really substantively differ in terms of out-of-pocket expenditures for patients; I find it difficult to imagine that forcing insurance companies to be nonprofit would do much to reduce costs.

> large insurance pools that must span age groups and risk groups.

What you describe (community rating) has been tried and it works. But it requires that a lot of young, healthy people enroll, and seniors receive most of the care. In an inverted demographic pyramid like most Western economies have, this is a ticking time bomb, so costs will continue to rise.

> Mandate at least decent minimal coverage standards

I think a better solution is to allow the government to threaten in negotiating prices with companies as Canada does; it greatly reduces rent-seeking behavior by pharmaceutical companies while allowing them to continue earning profits and innovating. (I understand a lot of the complaints against big pharma but they are actually one of the few sectors of the economy that doesn't park their wealth and actually uses it for substantive R&D, despite what the media will tell you, and countless lives have been saved because of pharma company profits)

Essentially the gist of what I'm saying, as someone who has been involved with and studied this industry for the better part of five years, is that it's much more complex than what meets the eye.

reply
bandrami
1 hour ago
[-]
There are a lot of not-for-profit insurance companies and they aren't noticably cheaper, though I'm not in HR and they may well be cheaper for the employer.
reply
kunai
1 hour ago
[-]
Disruption, yes, in the sense that the current system needs to be overhauled. But this is a space that's frequented by the SV and VC space and "disruption" has very different connotations, usually in the realm of thought that suggests some SV-brained solution to an existing problem. In some edge cases like Uber/Lyft, this upending of an existing market can yield substantial positive externalities for users. Other "heavy industry" adjacent sectors, not so much. Healthcare and aviation, not so much.

Even SpaceX's vaunted "disruption" is just clever resource allocation; despite their iterative approach to building rockets being truly novel they're not market disruptors in the same way SV usually talks about them. And their approach has some very obvious flaws relative to more traditional companies like BO, which as of now has a lower failure-to-success ratio.

I don't think you'll find many providers clamoring for an AI-assisted app that hallucinates nonexistent diseases, there are plenty of those already out there that draw the ire of many physicians. Where the industry needs to innovate is in the insurance space, which is responsible for the majority of costs, and the captive market and cartel behavior thereof means that this is a policy and government issue, not something that can be solved with rote Silicon Valley style startup-initiated disruption; that I would predict would quickly turn into dysfunction and eventual failure.

Enshittification has done a lot of damage to the concept of "disrupting" markets. It's DOA in risk-averse fields.

reply
miltonlost
2 hours ago
[-]
The part that needs disrupting is the billionaires who own insurance companies and demand profit from people's health .
reply
zdragnar
1 hour ago
[-]
The profit in insurance is the volume, not the margin. Disrupting it will not dramatically change outcomes, and will require changes to regulation, not business policy.
reply
kunai
1 hour ago
[-]
Agreed. I'd also argue that there will always be the issue of adverse selection, which in any system that doesn't mandate that all individuals be covered for healthcare regardless of risk profile, will continue to raise costs regardless of whether or not margins are good or bad. That dream died with the individual mandate, and if the nation moves even further away from universal healthcare, we will only see costs rise and not fall as companies shoulder more and more of the relative risk.
reply
eru
1 hour ago
[-]
Profit is a small part of overall revenue.
reply
Terretta
1 hour ago
[-]
> Google … constantly measures and reviews the quality of its summaries across many different categories of information, it added.

Notice how little this sentence says about whether anything is any good.

reply
ipython
2 hours ago
[-]
... at the same time, OpenAI launches their ChatGPT Health service: https://openai.com/index/introducing-chatgpt-health/, marketed as "a dedicated experience in ChatGPT designed for health and wellness."

So interesting to see the vastly different approaches to AI safety from all the frontier labs.

reply
_cs2017_
1 hour ago
[-]
Why vastly different?

Aren't they both searching various online sources for relevant information and feeding that into the LLM?

reply
leptons
2 hours ago
[-]
Good. I typed in a search for some medication I was taking and Google's "AI" summary was bordering on criminal. The WebMD site had the correct info, as did the manufacturer's website. Google hallucinated a bunch of stuff about it, and I knew then that they needed to put a stop to LLMs slopping about anything to do with health or medical info.
reply
chrisjj
2 hours ago
[-]
s/hallucinated/fabricated/, please.
reply
Terretta
1 hour ago
[-]
arguably: incorrectly guessed*

in a way, all overconfident guessing is a better match for the result than hallucination or fabrication would be

"confabulation", though, seems perfect:

“Confabulation is distinguished from lying as there is no intent to deceive and the person is unaware the information is false. Although individuals can present blatantly false information, confabulation can also seem to be coherent, internally consistent, and relatively normal.”

https://en.wikipedia.org/wiki/Confabulation

* insofar as “guess” conveys an attempt to be probably in the zone

reply
xnx
2 hours ago
[-]
Ars rips of this original reporting, but makes it worse by leaving out the word "some" from the title.

‘Dangerous and alarming’: Google removes some of its AI summaries after users’ health put at risk: https://www.theguardian.com/technology/2026/jan/11/google-ai...

reply
miltonlost
2 hours ago
[-]
Removing "some" doesn't make it worse. They didn't include "all" AI titles which it would. "Google removes AI health summaries after investigation finds dangerous flaws " is functionally equivalent to "Google removes some of its AI summaries after users’ health put at risk"

Oh, and also, the Ars article itself still contains the word "Some" (on my AB test). It's the headline on HN that left it out. So your complaint is entirely invalid: "Google removes some AI health summaries after investigation finds “dangerous” flaws"

reply
jeffbee
2 hours ago
[-]
Google is really wrecking its brand with the search AI summaries thing, which is unbelievably bad compared to their Gemini offerings, including the free one. The continued existence of it is baffling.
reply
benregenspan
55 minutes ago
[-]
It's mystifying. A relative showed me a heavily AI-generated video claiming a Tesla wheelchair was coming (self-driving of course, with a sub-$800 price tag). I tried to Google it to quickly debunk and got an AI Overview confidently stating it was a real thing. The source it linked to: that same YouTube video!
reply
gvedem
1 hour ago
[-]
Yeah. It's the final nail in the coffin of search, which now actively surfaces incorrect results when it isn't serving ads that usually deliberately pretend to be the site you're looking for. The only thing I use it for any more is to find a site I know exists but I don't know the URL of.
reply
nickphx
19 minutes ago
[-]
huh.. so google doesn't trust it's own product.. but openai and anthropic are happy to lie? lol
reply
SirIsaacGluten
40 minutes ago
[-]
How could they even offer that without a Medical Device license? where is the FDA when it comes to enforcement?
reply
bjourne
51 minutes ago
[-]
Google for "malay people acne" or other acne-related queries. It will readily spit out the dumbest pseudo science you can find. The AI bot finds a lot of dumb shit on the internet which it serves back to you on the Google page. You can also ask it about the Kangen MLM water scam. Why do athletes drink Kangen water? "Improved Recovery Time" Sure buddy.

Also try "health benefits of circumcision"...

reply
catlikesshrimp
26 minutes ago
[-]
I agree with your point.

Going offtopic: The "health benefits of circumcision" bogus has existed for decades. The search engines are returning the results of bogus information, because the topic is mostly relevant for its social and religious implications.

I am related with the topic, and discussion is similar to topics on politics: Most people don't care and will stay quiet while a very aggresive group will sell it as a panacea.

reply
jnamaya
2 hours ago
[-]
chatGPT told me, I am the healthiest guy in the world, and I believe it
reply