LiteLLM Python package compromised by supply-chain attack
169 points
1 hour ago
| 37 comments
| github.com
| HN
detente18
25 seconds ago
[-]
LiteLLM maintainer here, this is still an evolving situation, but here's what we know so far:

1. Looks like this originated from the trivvy used in our ci/cd - https://github.com/search?q=repo%3ABerriAI%2Flitellm%20trivy... https://ramimac.me/trivy-teampcp/#phase-09

2. If you're on the proxy docker, you were not impacted. We pin our versions in the requirements.txt

3. The package is in quarantine on pypi - this blocks all downloads.

We are investigating the issue, and seeing how we can harden things. I'm sorry for this.

reply
jFriedensreich
4 minutes ago
[-]
We just can't trust dependencies and dev setups. I wanted to say "anymore" but we never could. Dev containers were never good enough, too clumsy and too little isolation. We need to start working in full sandboxes with defence in depth that have real guardrails and UIs like vm isolation + container primitives and allow lists, egress filters, seccomp, gvisor and more but with much better usability. Its the same requirements we have for agent runtimes, lets use this momentum to make our dev environments safer! In such an environment the container would crash, we see the violations, delete it and dont' have to worry about it. We should treat this as an everyday possibility not as an isolated security incident.
reply
kalib_tweli
58 seconds ago
[-]
Would value your opinion on my project to isolate creds from the container:

https://github.com/calebfaruki/tightbeam https://github.com/calebfaruki/airlock

This is literally the thing I'm trying to protect against.

reply
shay_ker
21 minutes ago
[-]
A general question - how do frontier AI companies handle scenarios like this in their training data? If they train their models naively, then training data injection seems very possible and could make models silently pwn people.

Do the labs label code versions with an associated CVE to label them as compromised (telling the model what NOT to do)? Do they do adversarial RL environments to teach what's good/bad? I'm very curious since it's inevitable some pwned code ends up as training data no matter what.

reply
tomaskafka
14 minutes ago
[-]
Everyone’s (well, except Anthropic, they seem to have preserved a bit of taste) approach is the more data the better, so the databases of stolen content (erm, models) are memorizing crap.
reply
datadrivenangel
4 minutes ago
[-]
This was a compromise of the library owners github acccounts apparently, so this is not a related scenario to dangerous code in the training data.

I assume most labs don't do anything to deal with this, and just hope that it gets trained out because better code should be better rewarded in theory?

reply
Imustaskforhelp
14 minutes ago
[-]
I am pretty sure that such measures aren't taken by AI companies, though I may be wrong.
reply
alansaber
10 minutes ago
[-]
The API/online model inference definitely runs through some kind of edge safeguarding models which could do this.
reply
intothemild
35 minutes ago
[-]
I just installed Harness, and it instantly pegged my cpu.. i was lucky to see my processes before the system hard locked.

Basically it forkbombed `grep -r rpcuser\rpcpassword` processes trying to find cryptowallets or something. I saw that they spawned from harness, and killed it.

Got lucky, no backdoor installed here from what i could make out of the binary

reply
hmokiguess
4 minutes ago
[-]
What is Harness?
reply
mohsen1
1 minute ago
[-]
If it was not spinning so many Python processes and not overwhelming the system with those (friends found out this is consuming too much CPU from the fan noise!) it would have been much more successful. So similar to xz attack
reply
ramimac
32 minutes ago
[-]
This is tied to the TeamPCP activity over the last few weeks. I've been responding, and keeping an up to date timeline. I hope it might help folks catch up and contextualize this incident:

https://ramimac.me/trivy-teampcp/#phase-09

reply
rdevilla
28 minutes ago
[-]
It will only take one agent-led compromise to get some Claude-authored underhanded C into llvm or linux or something and then we will all finally need to reflect on trusting trust at last and forevermore.
reply
cozzyd
10 minutes ago
[-]
The only way to be safe is to constantly change internal API's so that LLM's are useless at kernel code
reply
Imustaskforhelp
17 minutes ago
[-]
If that would happen, The worry I would have is of all the sensitive Government servers from all over the world which might be then exploited and the amount of damage which can be caused silently by such a threat actor or something like AWS/GCP/these massive hyperscalers which are also used by the governments around the globe at times.

The possibilities within a good threat could be catastrophic if we assume so, and if we assume nation-states to be interested in sponsoring hacking attacks (which many nations already do) to attack enemy nations/gain leverage. We are looking at damage within Trillions at that point.

But I would assume that Linux might be safe for now, it might be the most looked at code and its definitely something safe.

LLVM might be a bit more interesting as it might go a little unnoticed but hopefully people who are working at LLVM are well funded/have enough funding to take a look at everything carefully to not have such a slip up.

reply
MuteXR
19 minutes ago
[-]
You know that people can already write backdoored code, right?
reply
ipython
15 minutes ago
[-]
But now you have compromise _at scale_. Before poor plebs like us had to artisinally craft every back door. Now we have a technology to automate that mundane exploitation process! Win!
reply
MuteXR
12 minutes ago
[-]
You still have a human who actually ends up reviewing the code, though. Now if the review was AI powered... (glances at openclaw)
reply
hiciu
57 minutes ago
[-]
Besides main issue here, and the owners account being possibly compromised as well, there's like 170+ low quality spam comments in there.

I would expect better spam detection system from GitHub. This is hardly acceptable.

reply
orf
44 minutes ago
[-]
i'm guessing it's accounts they have compromised with the stealer.
reply
tom_alexander
6 minutes ago
[-]
Only tangentially related: Is there some joke/meme I'm not aware of? The github comment thread is flooded with identical comments like "Thanks, that helped!", "Thanks for the tip!", and "This was the answer I was looking for."

Since they all seem positive, it doesn't seem like an attack but I thought the general etiquette for github issues was to use the emoji reactions to show support so the comment thread only contains substantive comments.

reply
incognito124
4 minutes ago
[-]
In the thread:

> It also seems that attacker is trying to stifle the discussion by spamming this with hundreds of comments. I recommend talking on hackernews if that might be the case.

reply
vultour
1 minute ago
[-]
These have been popping up on all the TeamPCP compromises lately
reply
nickvec
4 minutes ago
[-]
Ton of compromised accounts spamming the GH thread to prevent any substantive conversation from being had.
reply
jbkkd
4 minutes ago
[-]
Those are all bots commenting, and now exposing themselves as such.
reply
Imustaskforhelp
2 minutes ago
[-]
Bots to flood the discussion to prevent any actual conversation.
reply
bratao
1 hour ago
[-]
Look like the Founder and CTO account has been compromised. https://github.com/krrishdholakia
reply
jadamson
53 minutes ago
[-]
Most his recent commits are small edits claiming responsibility on behalf of "teampcp", which was the group behind the recent Trivy compromise:

https://news.ycombinator.com/item?id=47475888

reply
soco
35 minutes ago
[-]
I was just wondering why the Trivy compromise hit only npm packages, thinking that bigger stuff should appear sooner or later. Here we go...
reply
franktankbank
54 minutes ago
[-]
Or his company is trash and hes moved onto plain old theft.
reply
eoskx
16 minutes ago
[-]
This is bad, especially from a downstream dependency perspective. DSPy and CrewAI also import LiteLLM, so you could not be using LiteLLM as a gateway, but still importing it via those libraries for agents, etc.
reply
nickvec
12 minutes ago
[-]
Wow, the postmortem for this is going to be brutal. I wonder just how many people/orgs have been affected.
reply
eoskx
11 minutes ago
[-]
Yep, I think the worst impact is going to be from libraries that were using LiteLLM as just an upstream LLM provider library vs for a model gateway. Hopefully, CrewAI and DSPy can get on top of it soon.
reply
nickvec
25 minutes ago
[-]
Looks like all of the LiteLLM CEO’s public repos have been updated with the description “teampcp owns BerriAI” https://github.com/krrishdholakia
reply
cpburns2009
1 hour ago
[-]
reply
jbkkd
5 minutes ago
[-]
reply
xinayder
19 minutes ago
[-]
When something like this happens, do security researchers instantly contact the hosting companies to suspend or block the domains used by the attackers?
reply
redrove
5 minutes ago
[-]
First line of defense is the git host and artifact host scrape the malware clean (in this case GitHub and Pypi).

Domains might get added to a list for things like 1.1.1.2 but as you can imagine that has much smaller coverage, not everyone uses something like this in their DNS infra.

reply
0fflineuser
31 minutes ago
[-]
I was running it in my homelab with docker compose using the litellm/litellm:latest image https://hub.docker.com/layers/litellm/litellm/latest/images/... , I don't think this was compromised as it is from 6 months ago.

I guess I am lucky as I have watchower automatically update all my containers to the latest image every morning if there are new versions.

I also just added it to my homelab this sunday, I guess that's good timing haha.

reply
xunairah
9 minutes ago
[-]
Version 1.82.7 is also compromised. It doesn't have the pth file, but the payload is still in proxy/proxy_server.py.
reply
postalcoder
39 minutes ago
[-]
This is a brutal one. A ton of people use litellm as their gateway.
reply
eoskx
10 minutes ago
[-]
Not just as a gateway in a lot cases, but CrewAI and DSPy use it directly. DSPy uses it as its only way to call upstream LLM providers and CrewAI falls back to it if the OpenAI, Anthropic, etc. SDKs aren't available.
reply
Imustaskforhelp
28 minutes ago
[-]
Do you feel as if people will update litellm without looking at this discussion/maybe having it be automatic which would then lead to loss of crypto wallets/ especially AI Api keys?

Now I am not worried about the Ai Api keys having much damage but I am thinking of one step further and I am not sure how many of these corporations follow privacy policy and so perhaps someone more experienced can tell me but wouldn't these applications keep logs for legal purposes and those logs can contain sensitive information, both of businesses but also, private individuals perhaps too?

reply
daveguy
23 minutes ago
[-]
Maybe then people will start to realize crypto isn't even worth the stored bits.

Irrevocable transfers... What could go wrong?

reply
kevml
1 hour ago
[-]
reply
rgambee
42 minutes ago
[-]
Looking forward to a Veritasium video about this in the future, like the one they recently did about the xz backdoor.
reply
sschueller
39 minutes ago
[-]
Does anyone know a good alternate project that works similarly (share multipple LLMs across a set of users)? LiteLLM has been getting worse and trying to get me to upgrade to a paid version. I also had issues with creating tokens for other users etc.
reply
redrove
2 minutes ago
[-]
Bifrost is the only real alternative I'm aware of https://github.com/maximhq/bifrost
reply
sschueller
3 minutes ago
[-]
I just found https://github.com/jasmedia/InferXgate which looks interesting although quite new and not supporting so many providers.
reply
river_otter
23 minutes ago
[-]
github.com/mozilla-ai/any-llm :)
reply
tacoooooooo
25 minutes ago
[-]
pydantic-ai
reply
kstenerud
21 minutes ago
[-]
We need real sandboxing. Out-of-process sandboxing, not in-process. The attacks are only going to get worse.

That's why I'm building https://github.com/kstenerud/yoloai

reply
dec0dedab0de
18 minutes ago
[-]
github, pypi, npm, homebrew, cpan, etc etc. should adopt a multi-multi-factor authentication approach for releases. Maybe have it kick in as a requirement after X amount of monthly downloads.

Basically, have all releases require multi-factor auth from more than one person before they go live.

A single person being compromised either technically, or by being hit on the head with a wrench, should not be able to release something malicious that effects so many people.

reply
worksonmine
7 minutes ago
[-]
And how would that work for single maintainer projects?
reply
oncelearner
29 minutes ago
[-]
That's a bad supply-chain attack, many folks use litellm as main gateway
reply
rdevilla
25 minutes ago
[-]
laughs smugly in vimscript
reply
6thbit
33 minutes ago
[-]
Worth exploring safeguard for some: The automatic import can be suppressed using Python interpreter’s -S option.

This would also disable site import so not viable generically for everyone without testing.

reply
fratellobigio
29 minutes ago
[-]
It's been quarantined on PyPI
reply
0123456789ABCDE
17 minutes ago
[-]
airflow, dagster, dspy, unsloth.ai, polar
reply
gkfasdfasdf
35 minutes ago
[-]
Someone needs to go to prison for this.
reply
nickspacek
55 minutes ago
[-]
teampcp taking credit?

https://github.com/krrishdholakia/blockchain/commit/556f2db3...

  - # blockchain
  - Implements a skeleton framework of how to mine using blockchain, including the consensus algorithms.
  + teampcp owns BerriAI
reply
mikert89
38 minutes ago
[-]
Wow this is in a lot of software
reply
eoskx
12 minutes ago
[-]
Yep, DSPy and CrewAI have direct dependencies on it. DSPy uses it as its primary library for calling upstream LLM providers and CrewAI falls back to it I believe if the OpenAI, Anthropic, etc. SDKs aren't available.
reply
Imustaskforhelp
38 minutes ago
[-]
Our modern economy/software industry truly runs on egg-shells nowadays that engineers accounts are getting hacked to create a supply-chain attack all at the same time that threat actors are getting more advanced partially due to helps of LLM's.

First Trivy (which got compromised twice), now LiteLLM.

reply
cpburns2009
28 minutes ago
[-]
LiteLLM is now in quarantine on PyPI [1]. Looks like burning a recovery token was worth it.

[1]: https://pypi.org/project/litellm/

reply
6thbit
36 minutes ago
[-]
title is bit misleading.

The package was directly compromised, not “by supply chain attack”.

If you use the compromised package, your supply chain is compromised.

reply
iwhalen
1 hour ago
[-]
What is happening in this issue thread? Why are there 100+ satisfied slop comments?
reply
bakugo
54 minutes ago
[-]
Attackers trying to stifle discussion, they did the same for trivy: https://github.com/aquasecurity/trivy/discussions/10420
reply
Imustaskforhelp
40 minutes ago
[-]
I have created an comment to hopefully steer the discussion towards hackernews if the threat actor is stifling genuine comments in github by spamming that thread with 100's of accounts

https://github.com/BerriAI/litellm/issues/24512#issuecomment...

reply
nubg
59 minutes ago
[-]
Are they trying to slide stuff down? but it just bumps stuff up?
reply
cirego
1 hour ago
[-]
First thing I noticed too.
reply
kevml
1 hour ago
[-]
Potentially compromised?
reply
TZubiri
40 minutes ago
[-]
Thank you for posting this, interesting.

I hope that everyone's course of action will be uninstalling this package permanently, and avoiding the installation of packages similar to this.

In order to reduce supply chain risk not only does a vendor (even if gratis and OS) need to be evaluated, but the advantage it provides.

Exposing yourself to supply chain risk for an HTTP server dependency is natural. But exposing yourself for is-odd, or whatever this is, is not worth it.

Remember that you are programmers and you can just program, you don't need a framework, you are already using the API of an LLM provider, don't put a hat on a hat, don't get killed for nothing.

And even if you weren't using this specific dependency, check your deps, you might have shit like this in your requirements.txt and was merely saved by chance.

An additional note is that the dev will probably post a post-mortem, what was learned, how it was fixed, maybe downplay the thing. Ignore that, the only reasonable step after this is closing a repo, but there's no incentive to do that.

reply
xinayder
36 minutes ago
[-]
> Remember that you are programmers and you can just program, you don't need a framework, you are already using the API of an LLM provider, don't put a hat on a hat, don't get killed for nothing.

Programming for different LLM APIs is a hassle, this library made it easy by making one single API you call, and in the backstage it handled all the different API calls you need for different LLM providers.

reply
otabdeveloper4
19 minutes ago
[-]
There's only two different LLM APIs in practice (Anthropic and everyone else), and the differences are cosmetic.

This is like a couple hours of work even without vibe coding tools.

reply
circularfoyers
19 minutes ago
[-]
Comparing this project to is-odd seems very disingenuous to me. My understanding is this was the only way you could use llama.cpp with Claude Code for example, since llama.cpp doesn't support the Anthropic compatible endpoint and doing so yourself isn't anywhere near as trivial as your comparison. Happy to be corrected if I'm wrong.
reply
chillfox
35 minutes ago
[-]
Now I feel lucky that I switched to just using OpenRouter a year ago because LiteLLM was incredible flaky and kept causing outages.
reply
deep_noz
58 minutes ago
[-]
good i was too lazy to bump versions
reply
jadamson
45 minutes ago
[-]
In case you missed it, according to the OP, the previous point release (1.82.7) is also compromised.
reply
dot_treo
38 minutes ago
[-]
Yeah, that release has the base64 blob, but it didn't contain the pth file that auto triggers the malware on import.
reply
jadamson
34 minutes ago
[-]
The latest version with the the pth file doesn't require an import to trigger the exploit (just having the package installed is enough thanks to [1]).

The previous version triggers on `import litellm.proxy`

Again, all according to the issue OP.

[1] https://docs.python.org/3/library/site.html

reply
otabdeveloper4
23 minutes ago
[-]
LiteLLM is the second worst software project known to man. (First is LangChain. Third is OpenClaw.)

I'm sensing a pattern here, hmm.

reply
nickvec
20 minutes ago
[-]
Not familiar with LangChain besides at a surface level - what makes it the worst software project known to man?
reply
eoskx
14 minutes ago
[-]
LangChain at least has its own layer for upstream LLM provider calls, which means it isn't affected by this supply chain compromise. DSPy uses LiteLLM as its primary way to call OpenAI, etc. and CrewAI imports it, too, but I believe it prefers the vendor libraries directly before it falls back to LiteLLM.
reply