Dependabot version updates introduce default package cooldown
88 points
3 hours ago
| 9 comments
| github.blog
| HN
zihotki
2 hours ago
[-]
If everyone starts applying cooldowns, won't it postpone the problem? So now there is a considerable amount of users who are affected and someone from the affected group discovers the infection and reports it.

But if everyone will be delaying updates, won't be there less chances to catch it in time? I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.

reply
stusmall
2 hours ago
[-]
>If everyone starts applying cooldowns, won't it postpone the problem?

There are still research firms who are actively and aggressively scanning new packages once they are pushed. For example socket.dev pulls new packages across ecosystems and performs automated analysis and runs it in a sandbox. We don't have to have them go boom in someone's production repos to find out there is a problem.

reply
tharkun__
32 minutes ago
[-]
And if every malware developer worth their salt now introduces code to "wait out" that period of time, we're back to square one.

This assumes that they employ clandestine enough techniques that you have to actually install, wait and observe the behavior for longer than the cooldown period in order to detect this, because the code is "obfuscated" enough to evade static analysis of the code. It's anti-virus / anti-anti-virus 101 all over so to speak.

The good thing I suppose is that it raises the bar. Your regular "virus generator" script kid (sorry: supply chain attack generator script kid) can no longer pull this off.

reply
woodruffw
25 minutes ago
[-]
Fundamentally, this is a cat-and-mouse game. But I suspect that "time bomb" techniques aren't economically viable for attackers, at least not with current patterns: current attackers demonstrate "smash and grab" tendencies because they know their access is limited anyways. Attempting to wait out a cooldown exposes them to additional detection risk.

Of course, maybe the attacker profile changes over time. But that's the nature of the game.

reply
tedivm
21 minutes ago
[-]
Most automated analysis isn't dependent on just behavior, but rather suspicious things in the code itself. You have a popular open source package with files that exist on pypi but not github then that's a big flag, or if a similar package suddenly has some base64encoded garbage that runs through an obfuscated exec call. In other words the simple fact that the project has obfuscated code is enough to flag for further attention.

That said if the only issue is time, researchers will just run their automated analysis through machines with dates in the future alongside their normal tests.

reply
tharkun__
3 minutes ago
[-]
Cat and mouse like the sibling says ;) Like if you start changing system time, I'll keep a log of system time to detect any "jumps" and then "behave normally" if I detect this. Of course I'll run the code that does this through "my obfuscator".

The thing with cat and mouse based on time is that this now became a default. I rather liked my odds when malware authors assumed that the defaults were that dependabot updates right away. If the general consensus online seems to be 7 days, then I'll set my dependabot to wait 10 days, so on average I'll catch even things people report over a weekend. Now that the default is a longer time period, I have to change my time period to be even longer, which actually increases my risk in another way: I'll stay vulnerable to _actual_ vulnerabilities vs. supply chain attacks for longer.

reply
NewJazz
1 hour ago
[-]
Also as an upstream, if your "coworker" releases a strange package without discussing the changes with the broader maintenance group, you might notice after 3-48 hours, but probably not within the hour unless you happened to be online.
reply
brikym
5 minutes ago
[-]
When you're running from a bear on a hiking trip you just have to be faster than your friend. So just set your cooldown slightly longer than everyone else's cooldowns. The cooldown will give security researchers some time to scan the packages so it's still good.
reply
woodruffw
2 hours ago
[-]
> But if everyone will be delaying updates, won't be there less chances to catch it in time?

No: the security assumption behind cooldowns rests on security scanning parties, not on innocent users being victimized. Three days is a short cooldown, but it should be a good enough lead for scanning parties.

> I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.

It’s not that much data, particularly for parties that are directly financially incentivized to be the first to report malware.

reply
MeetingsBrowser
2 hours ago
[-]
Only a few of the recent supply chain attacks were discovered by users noticing weird behavior.

The majority were noticed by maintainers or third party groups noticing things like releases not tied to a source tag, many rapid releases, etc.

Cooldowns won’t stop everything, but it makes a malicious release significantly more likely to be noticed

reply
roblabla
2 hours ago
[-]
The goal is to give time for automated scanners ran by cybersecurity companies to flag malware before it gets installed on real users.
reply
ronbenton
2 hours ago
[-]
Easy, then you just delay your project’s dependency updates just a little more than everyone else
reply
tabwidth
2 hours ago
[-]
Most of the malicious ones just curl something in a postinstall script, scanners already catch that. The sneaky ones don't look malicious until they run, and three days may not help.
reply
MeetingsBrowser
1 hour ago
[-]
There are plenty of ways to notice a malicious release without observing it running.

Build provenance, maintainer alerts on new releases, tying releases to specific git tags, etc all help.

reply
drdexebtjl
1 hour ago
[-]
Every single one now will be more sneaky, and we’ll be operating on a 3-day cooldown for no reason.
reply
brookst
1 hour ago
[-]
You really think it has zero benefit whatsoever? Nothing malicious will be caught?
reply
oakesm9
2 hours ago
[-]
I think the idea is that it gives a bit of time for the companies which run automated scans of new versions to run through and detect any issues with new versions before users install them en-mass.
reply
TZubiri
56 minutes ago
[-]
There was a story about two men and a tiger

The men see the tiger, one scrambles to run and the other starts putting on their shoes

"Why are you putting on shoes? You'll never outrun the tiger"

"I don't need to, I just need to outrun you"

reply
noosphr
2 hours ago
[-]
Watching language package managers reinvent everything distribution package managers have been doing since the 90s has been as fun as watching crypto people reinvent financial regulation.
reply
woodruffw
2 hours ago
[-]
The publishing topology is pretty fundamentally different: the entire power (and danger) of language package managers is that anybody can publish, not just a privileged few.

(This cuts both ways: I’d say that distribution package managers have learned valuable lessons about what users actually want from language package managers. Learning is a good thing.)

reply
kibwen
2 hours ago
[-]
This comparison is tiresome. Distro package managers are curated, language package managers are not. They're serving completely different use cases; the former is the App Store, the latter is the web.
reply
bitmasher9
17 seconds ago
[-]
They are literally solving the same problem, it’s just that distro packages operate on a lower level and thus receive more scrutiny. There have been plenty of examples of poisoned Linux packages, both at the source level and at the package level.
reply
noosphr
2 hours ago
[-]
Give it two years.
reply
Grokify
15 minutes ago
[-]
This seems to be primarily an issue with a few specific package management solutions that have suffered SCA vulnerabilities recently, not generaly across the board.
reply
Waterluvian
2 hours ago
[-]
I really hate dependabot making generic security people at work so pushy about updates updates updates. They seem to just be dogmatic about whatever dependabot says, forcing churn even when the documented issues are clearly not relevant. I’m not sure how to handle it politically. I’m convinced that updating so much more often is worse, not better.
reply
JoshTriplett
1 hour ago
[-]
> I’m convinced that updating so much more often is worse, not better.

The issue of cooldowns aside (which is about delaying updates, not reducing their frequency): you're going to have the same set of problems when you update, whether you do it frequently or infrequently. The difference is that if you update frequently, you'll have a smaller set of updates (so it's easier to debug) and you'll have more opportunity to report issues upstream and fix them in a timely fashion.

It's the same underlying problem as CI and build time. Most people abandoned the concept of projects that take so long to build you can only do testing once a week, because CI that runs on every PR provides a much better experience. This is the same lesson applied to updates.

reply
Waterluvian
29 minutes ago
[-]
My logical read of the situation is that I end up making fewer overall changes if I end up upgrading a dependency once, not thrice, to a specific version. And the changes are their own source of risk.
reply
bluejellybean
2 hours ago
[-]
I'm in a similar camp, I dislike how often third-party package updates get pushed out, especially given the lack of serious inspection.

The reality is that each update is its own potential security issue and with supply chain attacks being all too frequent, it's not a panacea.

reply
cesarb
2 hours ago
[-]
> The reality is that each update is its own potential security issue

Even beyond security issues: each update is a new opportunity for breakage, not only from bugs in the third-party package, but also from unexpected dependencies on the third-party package's behavior.

reply
demosthanos
1 hour ago
[-]
I don't know if all ecosystems are as bad as node is, but the node ecosystem has terrible issue severity ranking which makes infosec squeamish for no reason.

Every week or so there's a new High+ "vulnerability" that gets published against our dependencies and I have to go look at it to confirm that it's yet another case of "it's possible for someone to give this dev-only tool a bad regex that would cause the test runner to OOM on that branch".

reply
esikich
1 hour ago
[-]
As a sysadmin I'm in the same boat. I've unfortunately never worked with security folks that seemed to have any sysadmin or dev experience. Whether or not this is universal, idk, and I have no idea what they are teaching in these security courses. But I'm tired of security teams telling me "you need to implement these 230 group policies this quarter" or whatever. They don't seem to grasp the externalities of a request like that and how much fucking work it is to vet, test, deploy, monitor, verify, etc. 80% of the time, they don't even know what they do or if it's even impactful for us.
reply
bunderbunder
2 hours ago
[-]
I’ve mainly handled it by pushing my team to be extremely conservative about what dependencies we take, especially if they pull in scads and scads of transitive dependencies.

This elegantly mitigates three problems in one go: update churn, dependency hell, and supply chain attack surface.

It also, frankly, tends to make the code easier to understand. I’m not a huge NIH person but I do have to say that a lot of packages these days tend to encourage ways of doing things that are unnecessarily complex. More than once I’ve replaced a dependency with homegrown code and reduced LOC in the same commit.

reply
333c
1 hour ago
[-]
I feel the same. So so so many "regular expression denial-of-service" issues at my last job that just didn't seem serious or often reachable.
reply
mook
2 hours ago
[-]
But updates to broken packages are still allowed: if a new version is pushed within the three days, it does not reset the cool-down. You just get a pull request to update to a known-bad version instead.
reply
ashu1461
2 hours ago
[-]
This makes me think whether npm (and other registries) should apply security requirements based on ecosystem impact. Example a package having millions of downloads can have special security measures enforced.
reply
madeofpalk
2 hours ago
[-]
What would be a security measure that should only be selectively enforced?
reply
ashu1461
49 minutes ago
[-]
There are few ideas which come to my mind, some might be far fetched, taking NPM as an example.

- Restricting packages with similar names as of popular packages restrict expres because express is a popular package.

- Imposing stricter 2FA checks on accounts of authors of these packages.

- Making sure that published packages don't have vulnerabilities and clear npm audit.

- Alerts in case these packages contain a dependency which is new / relatively new.

reply
toomuchtodo
2 hours ago
[-]
Higher cost (“Mythos” vs static code analysis) vulnerability scanning prior to successful merge to main branch or deployment as an artifact. As risk increases (popular code->greater exposure potential), increase automated, programmatic scrutiny on subject code to lower residual risk.

(application security and vulnerability management is a component of my work in financial services, thoughts and opinions always my own)

reply
insanitybit
3 hours ago
[-]
What a state of things where we have to fear installing software, and rely on vendors to scan things ahead of time, because our supply chain is such a mess and our tooling is so incapable of (and uninterested in) protecting us.
reply
Insimwytim
3 hours ago
[-]
You cannot call it a supply chain, if you have zero contractual relationships with the authors of the solutions you are using.

[1] https://news.ycombinator.com/item?id=44434355

reply
cryo32
2 hours ago
[-]
Oh that one really makes you think doesn’t it.
reply
stingraycharles
2 hours ago
[-]
I mean, that’s just arguing over whether or not the definition of “supply” implies “compensation”, which isn’t very interesting imho.

The grandparent’s point remains the same, the software ecosystem and its supply chain or however you want to call it is a hot mess.

reply
xgulfie
2 hours ago
[-]
Traditionally the term "supply chain" has implied a buyer/seller relationship
reply
stingraycharles
2 hours ago
[-]
I think that’s up to debate, and my point is that debating whether free software counts as “supply” or not is really not that interesting.
reply
insanitybit
43 minutes ago
[-]
So what?
reply
madeofpalk
2 hours ago
[-]
What would a solution to this look like?

What would it take to not fear installing software? This isn't a npm problem, its a computing problem in general. Spaces like this are generally pretty against any sort of restrictions or limitations being put on computers under the name of safety (see Manifest v3)

reply
dwoldrich
2 hours ago
[-]
For libraries, I like the Gnu Affero Public License[1]. If you run the library in software with that license, you have to publish all the source of the entire project that incorporates it.

No corporation could tolerate this, though, so the library vendor can negotiate a commercial license of their software for appropriate fees.

That said, corporations are not going to want to negotiate fees with 100's of vendors over constantly fluctuating dependencies in their software.

This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.

Similar to Amazon's Dynamo API, whatever the next big language/ecosystem is needs to be designed around _billing_ and automatic license management for # of deployments, seats, call volumes, etc.

[1] https://web.archive.org/web/20260712154038/https://www.gnu.o...

reply
skissane
59 minutes ago
[-]
> This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.

I don't think this idea is going to go anywhere.

If a package is available for free, on convenient licensing terms, developers will use it.

If you make them pay, many developers will prefer to just build it themselves. Coding agents make that easier than ever.

Buying a package involves a lot more paperwork – it needs to go through procurement – and introduces new risks, e.g. what if the vendor increases their prices

There are potential exceptions – software with really advanced algorithms (e.g. solvers for optimisation problems); safety critical software; software needing regulatory certification (e.g. there are some Australian government APIs they won't let you call unless you've hired an auditor to certify the software you are calling them with, and the relevant government agency has approved the auditor's report) – but those exceptions are relatively rare, and the existing solutions are arguably adequate to handle them

I also think it is different for packaged SaaS applications [0] because there the buyer isn't a developer, it is someone non-technical, and "use a coding agent to build it yourself" isn't within their comfort zone or risk appetite (at least, not yet).

[0] conflict of interest disclaimer: work for a SaaS vendor

reply
insanitybit
42 minutes ago
[-]
> What would a solution to this look like?

Sandboxing and auditing built into the software from the start. Browser Extensions solved this ages ago.

reply
jchw
2 hours ago
[-]
Manifest v3's actual motive was so shamelessly transparent that most of us just don't allow the "safety" argument for it to really be entertained. I don't have a suspension of disbelief rich enough to pretend I don't know.
reply
sunaookami
3 hours ago
[-]
No way to prevent this says only package manager where this regularly happens.
reply
insanitybit
42 minutes ago
[-]
Every major package manager is just as fucked.
reply
cadamsdotcom
2 hours ago
[-]
"We don't call 'em 0days any more, now we call 'em 3days"
reply
throwatdem12311
2 hours ago
[-]
More like 3-per-day amirite
reply
bstsb
2 hours ago
[-]
> The default applies only to version updates. Security updates still open immediately, so critical fixes are never delayed.

does this require a real vulnerability report, or CVE? if the package is compromised would they just be able to push a false "critical update" that bypasses this wait?

reply
MeetingsBrowser
1 hour ago
[-]
Requires a GitHub security advisory and

> Only advisories reviewed by GitHub trigger alerts.

From https://docs.github.com/en/code-security/concepts/supply-cha...

reply
doctorpangloss
40 minutes ago
[-]
The idiocy of cooldowns speaks for itself.
reply