1. Browser fingerprinting or ip bans. They used advanced fingerprint-shifting browsers and residential proxy ips.
2. Phone number 2FA. Significantly slowed legitimate user access but still didn't fully stop credential stuffers.
What did work:
3. rate limits and carefully tailored scripts that detected usage patterns and autobanned. Eventually they gave up on us guess wasn't worth the trouble. However I'm sure we lost a few legitimate users too in the process.
What I would try in the future:
- Passkeys as 2fa. Most browser automation platforms can't handle passkey auth inside a VM.
Don't you typically use that for valid users? As-in, you allow access when the fingerpint matches their existing fingerprint and when it doesn't you require additional information to be presented (i.e. security code).
So if somebody shifts their ip around they end up needing more information than just user+pass to login but somebody that doesn't (i.e. a normal person at home) does have the easy way to login.
I'm guessing this would be Firefox, possibly using in house extensions or userscripts designed to help further avoid fingerprinting?
In my experience they're generally detectable by mismatches in various attributes compared to the "real" browser whose user agent they are spoofing (though of course, the ground truth of adversarial detection is always hard to know for sure).
Credential Stuffing is (or at least was) a gigantic market, and it is one of the biggest headaches for the biggest pay-walled services, like Netflix, HBO, Prime, etc.
The people that made a living out of it were stuffing millions or billions of credentials (sourced from database leaks) in the most popular services, hoping to then sell the accounts for small amounts of money, like a dollar for a Netflix account with a 10-day warranty. It's a numbers game at heart with a substantial technical aspect, where you need to optimize your checker code to essentially send properly formatted requests that can't be intercepted and don't arouse suspicion, and then you had an ecosystem of "methods" that are certain request-response chains that make your login request look like it's from a real person. People needed to figure out advanced methods to not invoke a CAPTCHA check, which is cost-prohibitive, but not impossible to solve automatically (AI wasn't a thing back then). You then have to buy millions of proxies that are able to route the requests from different IPs so that you're not sending millions of requests from a single IP. Checkers had reached a point where, depending on your proxies, were performing 10,000 or even 20,000 checks per minute. Multithreading was the cornerstone of these technologies, as a simple 2vCPU VM was already bottlenecked by proxy speeds.
Back when I looked into it, it was the wild west, as SSO and other technologies just weren't a thing yet. Companies would become fads of this credential stuffing scene, and it would take a dev team an entire sprint just for them to make a login page that was able to at least force a CAPTCHA check for each single request, and that's IF they had the proper monitoring tools to notice the gigantic spike in login requests. Having a valid account to a service like Ebay where you can then order whatever you want with the linked credit-card, you can understand how big of a security issue this is.
I haven't looked at it recently, but I assume that this has become vastly more difficult for the common-place services like streaming providers and digital goods marketplaces. SSO, IAM platforms like Keycloak, and advanced request scanning techniques have evolved. I'm guessing things have become substantially better, but it's always going to be a big issue for those smaller websites without a dedicated dev team or without at least someone maintaining them.
Then the fun thing was that some lawyers concluded this is still a breach on success and that we should be responsible and report/mitigate these.
How? How do you stop your users from making dumb decisions? The only solution seems to be to "give up" and go passwordless, putting the credentials to the big boys in town.
Best companies to work with were spycloud.com and sift.com.
spycloud actually specializes in identifying leaked credentials, which are what attackers use in the credential stuffing list they go through, so you could identify "stuffable" credentials prior to the attack happening, which is nice.
sift was great at helping to just identify fraud in general, so if an account did quietly get compromised, we could identify it before the transaction was finalized.
For the user it's kind of a a soft MFA via email where they don't have to enable it, but also don't always get the challenge.
Astonishingly, we had barely any complaints about the system via customer care and also didn't notice a drop in (valid) logins or conversion rates.
I tend to generate my passphrases for sites now, my only complaint is a password field should accept at least 100 characters. Assuming it's salted+hashed anyway, it's almost irresponsible to limit to under 20 characters. I'd rather see a minimum of 15 chars and a suggestion to use a "phrase or short sentence" in the hint/tip.
I wrote an auth system and integrated the zxcvbn strength check and HIBP as default enabled options. The password entry allowed for up to 1kb input, mostly as a practical limit. I also tend to prefer having auth separated from the apps, in that if auth fails via DDoS, etc, then already authenticated users aren't interrupted.
There was recently a bug in bcrypt implementation where characters after first 64 were silently ignored.
Anyway, while it is easy to require long password it is almost impossible to detect password reuse. The only way to solve the issue is to not let users to choose passwords, if they want to change it then generate a new one for them. And that isn't happening unless sites are forced to do it by government.
I think there are plenty of other solutions, including 2fa, push notifications and likely more valuable than any of the previous mentioned bits would be to ensure that SSO works across an organization.
In general, simply requiring a minimum length of say 15 chars and the suggestion to use a phrase or sentence is enough. I've switched Bitwarden to the word generation option with capitals and numbers, which usually works, except when there's an arbitrarily small maximum length on the input field.
I switched because trying to type 20 random characters including special characters in under 20s (was a remote terminal limit on a VM I'd misconfigured and had no other way in) was pretty much impossible and had to run the reimage from scratch.
I work for Stytch (another CIAM provider) on the fraud and security side and we do these too. I'd say you see credential stuffing defenses integrated into the auth provider rather than standalone rate limiting because so much of the relevant context is tied up in the auth side.
And, all the error messages end up being bad, as is the case for many security things. For our own features like Intelligent Rate Limiting https://stytch.com/docs/fraud/guides/device-fingerprinting/d... it's usually a bad idea to tell a user "You hit the limit, come back in an hour or contact support" because it gives an attacker information on how to improve. And we regularly see probing behavior where an attacker is trying to find the edges of a defense before starting a full-scale attack.
On the side topic of error messages - if you've ever seen "If your account exists, the password has been reset" that's another useless error message because "No account exists with that email" enables account enumeration.
Love how the HN community is sharing rate limiting and other credential stuffing "war stories" here too.
It wasn't that complex, but pretty nice in that it had pretty typical features for auth, was simple to setup/configure and would simply generate a signed jwt passed to your app/url. I had db adapters to be able to use SQLite, MS-SQL, PostgreSQL and DynamoDB. It also had integrations for AD, Okta and Azure Entra.
I guess ChatGPT doesn't know that one, yet.
That would rub me the wrong way for sure. Maybe not to the point of abandoning your platform, but I'd still be irked.
But this is worth it for certain platforms, especially where your users have something real to lose when they get pwned by cred stuffing (e.g. bitcoin casino).
Personally, I'd go a step further and say it's also worth it anywhere an account takeover can impact other people like Reddit and Twitter where cred stuffing lets you take over high profile accounts and then post a crypto scam or something more targeted, for example.
Mild inconvenience on a /register form doesn't sound like a serious counterbalance to the large cost of trivial account takeovers.
How are you securely delivering the password to the end user? How are they securely storing a string you created? One that is presumably high-entropy?
Or are you talking about one time use passwords (OTP)?