They were using AWS, so I logged in the account to add a few more machines. Right there, in front of my eyes, were the signs of an adversarial, abusive relationship.
The UI to fire up a new machine did not show me the price. I had to look up the price in another table that did not have the specs.
I had to have the two tables open, cross check the specs and price.
If I had learned one thing from my past life was that if you see the signs of an abusive relationship, you have the option to walk out, and you don't, all that follows is your own fault.
Created a DigitalOcean account, moved everything over. Set up our CI/CDs to deploy there, and spent the next two months on the product, launching one month earlier than promised.
Some years before that I saw a video online where a person digs a hole near a river and puts a pipe connecting the river and the hole. The fishes push themselves hard in the pipe to get to their trap. Choosing the path of least resistance, and never backing off from a mistake: recipes to end up like those fishes. The video left a big impression on me.
I remember many years ago we hired a junior developer who just finished his internship at AWS and he showed me the dashboard he shipped all by himself in the summer with no product or designer help. It looked horrible.
Some devs have a good product/UX sense but the vast majority are horrendously bad at UX.
My point is that maybe it was intentional, but just bad UX culture.
Edit: It wasn't intentional
Some background. I work at an Amazon sub. This is a good UI for the way we work. We don't spin up a single machine pretty much ever unless it's a cloud dev machine, at which point the price is listed at startup on a custom internal UI. They should consider putting that UI in the ec2 console.
When I spin up machines I pick an instance class by looking through specs and the price chart and set it via AI into a cdk construct. Usually pick a relatively normal machine type digging through all the ilvarious enterprise discounts (which are not reflectedin the prices in the console). Then as I roll out or when I get resource limit alarms on the fleet I adjust the instance types. Or when accounting asks me about price. In those cases I usually look if it's worth it to optimize.
The enterprise discounts are a big consideration. Every year new hires make bad decisions because they don't know about the discounts. They wildly affect total cost. Some things are more expensive (lambda first few years), and others are very cheap so we dog food. The console price in no way reflects reality.
In 15 years we've had about 1k services stood up, around 700 are active. 2000 or total counting tutorials and tests. That means out of an eng org of 500, we've made those decisions maybe 10k times total.
That's how Amazon thinks about it as well. So yeah I agree that the UI isn't meant to be like one where your spinning up a host. I haven't spun up a single host in like 5 years, but I've made many clusters.
But that doesn't mean it shouldn't be better to work for a wider audience. Customer obsession and all
This may be valid, but even if it is someone (or a group of people) at Amazon are violating one of their core leadership principles - Customer Obsession
https://www.amazon.jobs/content/en/our-workplace/leadership-...
A useful (and hopefully delightful) UX is key to showing customer obsession.
That being said, I personally feel the UX at Amazon sucks overall, not just for pricing/packaging but even getting basic shit done. So perhaps Amazon (or at least AWS) doesn't think a good UX is a key ingredient to demonstrating Customer Obsession.
AWS services names are notoriously bad at communicating what they actually do: https://expeditedsecurity.com/aws-in-plain-english/
Ask me how I know
In the end, our leadership changed what we were building so often that all of the UI work was scrapped long before we shipped. We ended up launching a janky console, quickly assembled by SDEs who were racing against deadlines. We skipped virtually all operational readiness work to meet the launch deadline. After claiming the launch win, the director, two managers, and the pm promptly left for other orgs.
I think that applies both to Amazon's dev system and pricing system. From what I hear about the insides, alignment is chaotic neutral inside of Amazon, but that shouldn't affect how we judge the system itself.
Often I see something that's supposed to be leaner - like Fargate is leaner than renting a whole server to run docker, right?
So it's cheaper as well? - Well, no.
Also if you reach any appreciable level of complexity, you should move to IaC - configuring all that stuff on the UI, and getting it right is torture.
"Azure’s Security Vulnerabilities Are Out of Control" - https://www.lastweekinaws.com/blog/azures_vulnerabilities_ar...
I just read:
> If I had learned one thing from my past life was that if you see the signs of an abusive relationship, you have the option to walk out, and you don't, all that follows is your own fault.
so... :)
if you see the signs of an abusive relationship, you have the option to walk out, and you don't, all that follows is your own fault.
Ignores nothing, and blames no victim.
It advises people to avoid becoming one when possible.
If it bothers you that you need to open two tabs for cross-checking the costs, you may want to avoid every cloud provider, not just AWS.
Once you have NAT gateways, CloudFront, S3, auto scaling, loadbalancers, etc, calculating the cost becomes an art rather than an exact science. And if you don't use these, there is no point of using AWS, there are plenty of "cheap" VPS providers.
You might have leftover reserve instance that applies, which make the listed price inaccurate. That reservation might even be in a different AWS account in the same organization that you don't have access to. That reservation might not even be there between the time you quote and the time you actually launch it if someone/something did launch before you.
Your organization might also have discounts. I believe some discount may also be very confidential. For example, my reseller policy is that the customer must not be able to see AWS Billing in the organization root account as supposedly the price in that console are the price AWS charged the reseller, while we pay listed price minus any discount we negotiated ourselves.
Finally, I suppose they don't want to have prices shown in multiple places as they will need to update it when prices changes. Doesn't want to risk forgetting one place and getting sued for it. You can see that AWS documentations often do not want to mention the price at all, even if that price is currently free.
Chinese clouds kinda make this simple by making reservation part of the buying machine itself - you have to mark that particular machine as monthly/yearly committed when you start it (or convert it later). The complicated part is recycling instances - if you delete a server before its reservation ends it ends up in a recycle bin that you need to look before making new reservations.
They cannot predict what my bandwidth consumption will be, or other such variable costs. For those, they tell you rates.
That's just for ec2. Everything is like this. Super awesome when you're being brought onto a new project and trying to estimate costs for your client. And let's not forget the little tiny things that should cost nothing. A NAT gateway with no redundancy is $30/mo. That's a fun surprise.
>AWS is not built around hobbyist needs
Yes, as if no startup teams are tasked to remain within hard spending targets when they're trying to build a POC with technologies that they are not initially experts in.
I think about the diversity in usage patterns: from generating giant video stream broadcast somebody trying to calculate yet another digit of pi. It’s wild.
Is true, probably, that AWS doesn’t know how much anyone’s use case will cost (even when it’s yet another version of something we’ve seen before). Too many variable.
If only there were some kind of software with a text based, natural language interface that we could ask a question like “how much would it cost to do XYNZ on AWS?”
Yes, as long as you do not have seasonal traffic, auto-scaling, spot instances, burstable instances, saving plans, reserved instances, floor/custom pricing, etc. These are tools to optimize your spendings and spend less if you know what you are doing.
> defending deliberately obfuscated pricing
A bit contradictory that price simulators are fine, but then the pricing is deliberately obfuscated. Then which one?
AWS is almost never required and almost never the best option. It's the Cisco of options, it's often the default but for no good reason other than someone on the team probably knows enough about AWS to make it work.
Almost every startup I've worked at has leveraged AWS as their primary but when not they end up using AWS for something. And in every startup there's always contention with AWS spend and all of these startups invest significant time and, funny enough money (via cost savings products or consulting), to reduce their AWS bill. And yet, they never seem to try anything else. Doomed to the cyclical cost savings cycle. Amazon knows this and the UI/UX is designed to keep companies in this money burning loop.
Finally, AWS isn't a silver bullet. For anyone in us-east-1, you know [0].
[0] https://mashable.com/article/amazon-web-services-outage-may-...
On Google cloud compute, the ui shows an updated 'cost' as you start building your machine.
No thank you
I’ve never felt surprised by pricing. Cost has been surprising, but that happens when usage is surprising in my experience.
They absolutely could to you a base price on the ec2 setup page, but they don't. And I have been absolutely surprised by pricing. Services that do almost nothing could cost more than your ec2.
I spent 5 years optimizing spendings on AWS at various companies. Yes, it does come with traps and footguns. On the other hand if you know what are you doing, there are plenty of tools to optimize your spendings with RIs, saving plans, auto-scaling, etc, and spend less than the list prices.
Based on my experience AWS for the companies that can afford to pay surprise bills out of pocket if something goes wrong.
This is false. The price shows up right away when you select a machine. I dont work for AWS...
> The UI to fire up a new machine did not show me the price. I had to look up the price in another table that did not have the specs.
I don’t want to be the one defending AWS, but I don’t think that this is a valid reason not to like them. I mean, pricing depends on so many factors like reserved/dedicated/spot/on-demand instances have all different prices.
I don’t even think that using the UI to spin up the machine is the right way to do that in an enterprise setting, you should always do that through Infrastructure as Code, to know exactly what you have up and running, just by looking at that as you would with any program. I’d suggest to use the UI for simple testing, for which the costs are often (but not always) negligible.
Jeff Bezos if you see this please send me some cash.
About using IaaC to set-up the infrastructure, sure, but sometimes you just need to browse stuff before actually writing code to get a feel.
Let’s look at Lambda for a second. Deploying a lambda function to AWS costs literally nothing. And yet, depending on how it’s used, it can cost an infinite amount of money. Which price should it show?
There are far more sevices like Lambda than EC2.
But that's the problem: The complexity of doing that properly is pretty much the same as just doing your own hardware (which is what I'm working with most of the time - handling stuff on physical servers). And at that point the question should be why you're paying AWS so much money and pay your people to automate AWS workflows when you could just pay them to automate workflows on physical hardware, which would be way cheaper to run than the AWS instances.
Or you can have your own negotiated private pricing which is a whole different story in itself.
If they know how to bill you then they obviously know how to consider and calculate all of these factors, they just choose not to show you up front.
Heck, I even have a hard time telling the price I pay on an account by account basis; because we have savings plans, those get charged against the root account and then I see $0 spent on EC2 in the individual account because it's all covered with a savings plan.
And when I'm putting together that IaC and trying to decide which new instance type to upgrade to, I have to dig through multiple confusing interfaces to figure out that what I want is to upgrade from m8a.4xlarge to c8a.8xlarge and how much that is going to cost me.
i just use vantage (https://instances.vantage.sh/) now. their api is functional and reasonable.
It should really be a read-only layer for metadata and logs.
I'm tired of people acting like complex infrastructure tooling is adversarial because it's not completely intuitive. Infrastructure is hard. AWS can give you tooling and docs with patterns to follow, but they can't read your mind. Neither can the PaaS providers - they just make choices on your behalf and hope it won't matter to you.
This is still hugely prevalent at some of the largest companies in the world
I get to see how a lot of companies use AWS. The console does make its appearances, but less and less often these days.
AWS takes as long as possible (for me it was a month) to respond to the initial DTO request, then require you to submit a multi-page form answering a barrage of questions about why you're leaving, where you're going to, what services you used, and estimated data egress. A week or so later, if they approve the request, you're not allowed to begin DTO until 60 days after the approval.
By the time you can egress your data for "free", you've been stuck on AWS for 3-4 months since you first made the decision to leave.
[1] https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i...
Well guess what, if you have a CRUD website and 100 users you're just not the target. Move on.
Some days ago I wanted to sketch a 3D model of my TV remote. I opened blender and what a mess of complicated windows and panes. I closed it immediatly. Do I think Blender is an over complicated mess? No, I just think I'm not the target. And I'm not offended to be too noob to use it.
It should be made clear though, that some of us helped spend many millions in obviously wasteful on-prem infra in the nineties, bought into AWS wholeheartedly when it came out, fought through the ignorance, developed the ability to deliver highly scaled applications on the platform over many years and at least some of us still carry those same beliefs:
- It's more complicated than it needs to be
- It's more expensive than it should be
- Pricing is more opaque than it should be
Meanwhile, the cost of other options (including self-managed, on-prem infra) has fallen massively since those early days of AWS.
This is completely backwards, at least with OpenSearch and Valkey. AWS didn't create the forks until after the upstream projects changed their license, so it's really weird to say that the forks "resulted" in the license changes when those forks where a response to the license changes. With Valkey in particular it was members of the former redis core development team that created Valkey.
Why do we apply this standard to MongoDB but not to Apache, Linux, Postgres, or MariaDB? One purpose of an open source license is to allow many providers to provide the service. As I've talked about here previously, Elasticsearch wasn't able to provide the service I needed, so I had to move to AWS.
It's weird to me that the Hacker News community doesn't think that sort of competition is good. The narrative seems to be that all these businesses are somehow victims of AWS, when it seems the truth is much more straightforward: they provided open source software and people used it. The fact that their business had no working plan to actually monetize that foundation should not be taken out on the community.
Many support breaking up Amazon so others could compete not killing small entities and growing Amazon.
The only people with any justification for hurt feelings are the community contributors.
They took software that others gave away for free without restriction and did what they wanted with it. It took time but the community figured out this exploit path and patched it in subsequent license versions.
Free Software was designed to avoid this, and has become stricter as the technology changed. Open Source was deliberately designed to thwart this. The entire intention of it was to allow businesses to resell work that was done for free. When you fork Free Software, your fork is also Free Software.
i’m only familiar with the postgres side, but it seems like a more nuanced view of this debate would be to discuss aws monetizing open source relative to their upstream, community-beneficial contributions.
Just try a little bit of understanding.
https://www.eff.org/deeplinks/2019/06/felony-contempt-busine...
We are supportive of 3rd party ink cartridges, and there's little concern for the business model of the printer manufacturers. We instead care about the rights of the folks using the printers.
With Postgres, no one bats an eye that there are thousands of hosting companies providing Postgres as an offering, and they give nothing back to the project. Same with Apache, Nextcloud, Linux, Nginx, Sqlite, and thousands of other pieces of open-source software. Are folks against hosting companies like https://yunohost.org/?
It's only when (1) the software is open-source, and (2) the entity behind it doesn't know how to sustain itself with open-source, that we suddenly change positions and view the project as a victim. This doesn't happen with printers, it doesn't happen with other open source software. I'm not even against a change in the license, but claiming that AWS is evil for doing this doesn't track.
Also, many of them predate hyperscalers and developed governance/economic structures that make them harder for AWS to capture or destabilize, whereas AWS free-riding a vendor-controlled project can destroy the economic engine sustaining the project itself.
Quite ironically, the only example from your list that doesn't predate hyperscalers (Nextcloud) is fundamentally a self-hosting/federation product. It exists largely as an alternative to hyperscaler-native platforms, not as a cloud primitive AWS can easily commoditise into its own stack.
So, treating PostgreSQL, Linux, Elasticsearch and Nextcloud as interchangeable "open source projects" ignores the completely different institutional and economic realities behind the projects.
Usually a community can tolerate changes only when it's not already near the bottom. When you're near the bottom, almost any destabilisation can kill your little system.
[0] https://www.fox32chicago.com/news/aldi-closes-west-pullman-c...
Even if it weren't AWS, someone else with enough determination could use the same open source code to create a compelling alternative taking away business from the original authors. Trying to use social norms to make people not do that is not effective. You need mechanisms that can be enforced via legal procedures to be effective.
Then why did they advertise themselves as open-source efforts when they weren't? They should have been the best possible providers of managed service offerings given they wrote the software they'd be managing, no?
Why are monopolies OK here but not elsewhere? Choosing a hard-to-win business model is not supposed to be a choice that guarantees you business income.
Maybe the business model / community-governance model does matter after all...
But those license changes were a response to how AWS was monetizing their work in ways unsustainable for the upstream projects.
Or seen from the other side, these projects chose initial licenses that didn't fit with their wants for how others should use their project, in this mind.
If you use a license that gives people the freedom to host your project as a service and make money that way, without paying you, and your goal was to make money that specific way, it kind of feels like you chose the wrong license here.
What was unsustainable (considering this perspective) was less that outside actors did what they were allowed to do, and more that they chose a license that was incompatible with their actual goals.
Since they're a for profit entity, they'll do whatever they think offers the best cost/benefit.
But it’s ok to be voluntarily grateful for hard work.
I can't just translate Harry Potter to Spanish and sell it.
Next, upstream responded with a license change, then amazon escalated with the fork.
But in context, it means “cloned/downloaded and offered as a hosted service.”
The fork came later, after the defensive license, which was in response to the clone+monetized hosting, eg ElasticSearch.
And I still remember JBoss and ahole Marc Fleury ...
Redis was not an open source company when AWS moved to Valkey.
Companies are free to license under the AGPL if they want. Or other open source licenses.
Sorry, but non-open source companies aren't getting sympathy from me because they are hating on open source projects.
That original license probably helped them with goodwill and to gain a community; when those benefits no longer exceeded the downsides of using that license, they changed licenses to one that suited them better.
Naturally, this change costs them some amount of goodwill, a portion of the very goodwill that they harvested by choosing an open-source license in the first place.
If you leave some apples at the side of the road, with a sign "$1 per apple" or whatever, and people largely pay enough for you to continue to pick apples, that's great. If someone starts coming every day and taking the entire crate, I don't blame you for discontinuing the convenient apple sales, I blame the thief.
It’s like someone said “free whole apples, or $2/lb for sliced apples.”
And someone came, took all the whole apples, cut them, and sold them themselves.
Let's be pedantic, and say someone gave apples away in exchange for donations, and when everyone only got a few apples and donated, things are fine, but then someone decided they can just take all the apples and sell them elsewhere.
Is it the fault of the first guy for not offering free apples any more, or is the second guy why we can't have nice things?
IAM is just complex. I can't think of any implementation of "users, groups, roles, policies, identity providers, oidc" that is truly simple.
I'm reminded of a guy I worked with, who fought against Kubernetes adoption because it was "too complex", only to slowly reinvent Kubernetes badly, adhoc, out of vault, consul, systemd, nomad, iscsi, ansible, jenkins, puppet, bash, spit, glue... making lots of mistakes along the way. You think you don't need to implement some feature until you do.
Another thing I'll say about AWS (having been the sole infra guy at a few startups) is that it's well within most people's abilities to learn it. And you can usually avoid the shitty stuff. You think lambdas stuck? Don't use them! You could use EKS, ECS or bare EC2.
IAM is great because it applies internally just like it does externally. The internal AWS team don't get more access than you do, and if we get access to do certain thing on your account to perform specific service that's because you have a service principle in your IAM trust relationship that allowed us access, that you can see, and audit. For instance, lambdas have lambda role because you don't want lambda service just reading your S3 buckets because "we're AWS we automatically get access", you can absolutely see and control access, even if it is internal to AWS.
That’s why it’s so complicated!!!
I don’t understand how I should evaluate trust for your internal EBS org versus your internal ALB org.
I kinda just expect it to be all “AWS” trust.
And it’s all garbage anyway. There’s no way I can prevent the hypothetically untrustworthy EBS team from surreptitiously adding charges to my account if they want to. Right? This would maybe make some sense if I could top level turn off/on services, but that isn’t how it works.
—
I have no doubt this makes some sense from someone inside the machine, but from the outside it’s not helpful nor useful.
The console kept warning me that I was giving root AWS access to my external application because they want people to use the locked in AWS path, and I was running off cloud.
On top of that, they break copy paste on the web console, so you can’t just ctrl-c ctrl-v and then ask Claude to explain their WTF-ery. Instead, you have to OCR or send a PNG.
I honestly did not think they could make IAM worse, yet here we are. Bastards.
This is a surprisingly common pattern in technology and software. Some things are definitively the “standard” at this point yet so many people simply refuse to spend the time to properly learn them.
It is also a surprisingly common pattern to adopt very complicated solutions for applications that are never going to need them
ultimately it is not possible to come up with a "standard" that is an acceptable replacement for good judgement
And that includes engkneers that only know how to use AWS and are terrified at having to learn something else.
1. Vercel Phase My first project used Vercel. Since my project was Next.js, the experience was decent. But as my project gained some users, I found that even for projects under 100 users, I needed to pay $20 per month. Since my service didn't require high performance, this cost felt steep.
2. Self-host Phase (Hetzner + Coolify) Later, I started setting up my own server with Hetzner and deploying with Coolify. Since Coolify is open-source and free, I only had to cover the cost of a VPS (even $5 a month was sufficient). I could deploy PostgreSQL instances and run a web server on it. But later I discovered that even this way, I still had to spend a lot of effort maintaining PostgreSQL and Redis. Even though they were containerized with Docker, managing them was still troublesome. I needed to pass various system and environment variables between services, which was very tedious.
3. Cloudflare Phase So later I switched to Cloudflare. With Cloudflare Workers, I can deploy fullstack applications and use D1 Database and Cloudflare KV to replace Redis. These features can be called directly within the Worker without needing to pass environment variables.
Plus, the local development experience is excellent and the pricing is very reasonable, so I've been using Cloudflare's entire suite ever since.
I think a mix of 2. and 3. is good for a small team or solo dev. Im throwing in a bit of homelab as well by adding some action runners and models on my desktop as well.
But cloudflare is great value for small teams. Not sure how it as at higher scale.
On the topic of env and config. It took me a while to get this write, and maybe overengineered.
But I invested a lot of time in trying to standardize env definitions, secrets manager, and per env config definition defined in my nx projects, and consumed by the commands or deployers. As well as pulumi for IaC.
I tried a couple of different approaches, but finally I just decided to use typescript as my config language. I use nx project.json but defined using typescript. And just define the env config as typescript functions to be injected to each command or deployment as a pure function of target env.
Here are a couple reasons of mine (PS I'm still a little new)
1) V8 isolates for serverless functions to address cold start problems, sure the entire node env ain't there but libraries like Hono are designed to work in that env... Combine that with their near immediate start-up - simple lovely
2) UI, AWS to me feels soulless, like if there's an entire industry to make AWS UI not suck it's obvious their UI is just bad, upto the point where people pay a premium for a good UI. Cloudflare UI is so much nicer, atleast to me
I recently developed a library and for that I made a landing page and documentation with Astro (no server just static stuff), and I was checking out how to deploy this and Vercel and Cloudflare, Vercel had a 100Gb/ month of bandwidth free which is nice, what's even nicer is cloudflare has infinite (practical infinite not the theoretical infinite ofcourse)
And once again, that's just lovely to work with!
It seemed like the bindings you needed to set to allow email can't actually be set (or even seen once Wrangler sets them) from the console at all.
Did docker make it easier?
The only issue I have with PostgreSQL is a bit of migration effort moving to new major versions.
> I needed to pass various system and environment variables between services, which was very tedious.
Was docker making this harder?
1. you have a limited number of global supported indexes, 5 iirc, which means your queries are very limited. If your use case ever expands beyond that you're pretty screwed. 2. You will have race conditions. Strong consistency is 2x the cost, and not supported on global indexes. 3. Data is split into 10GB partitions and all the read/write quotas are split evenly by the number of partitions. 100 reads you're paying for is actually 10 reads per partition if you have 10 partition. Hot sharding becomes a real problem.
Take your document data, stick it in a JSONB and you get the same performance way cheaper + query able/indexable columns. The only time Dynamo wins I think is it scales well globally, but you probably dont need it
The best way I can come up with to rack up a $75 bill for some prototype code is to vibe-code a thing that attempts to treat it like a SQL database with JOINs and GROUP BYs etc. Or similarly write code against it absent-mindedly with about as much understanding as a 2-year-old free AI tool.
Where it really shines is use-cases like I need like 1 or 2 simple relatively small tables of persistent storage and don't want to deal with a full RDBMS. Or I need 1 ridiculously huge table to be queried in a relatively simple way, and don't want to deal with fitting that data into a RDBMS.
I built an app a few years ago and needed some sort of DB to store around 50 million records that had ~10k reads+writes per month with 1 index. It cost me something like $50 to load it up initially, and then something stupid like 10 cents/month to maintain.
I would not build in DynamoDB if you suspect your access patterns will drastically change over the lifetime of the application (or if you intend to, e.g., plan to build a data warehouse or something crazy with it).
- It felt like far too much complexity just to do simple things.
- The obvious attempts to trap customers with slightly incompatible, higher level services felt gross
- The inability to run AWS trash on a dev machine had a MASSIVE hit on productivity
- Pricing didn't fall as fast as I felt it should (an obviously debatable position that reasonable, smart folks disagree with)
In my current company, we've been running basic SMB/tech startup functions on-prem (ACK! THE HORROR!) from ~6 basic computers (4 game machines and 2 nucs) for a few years now.
We just reconstituted the entire infra working part-time over about 2 weeks using Claude code and ansible.
It really doesn't make sense in this world to pay tens of thousands of dollars to rent a level of computation that can be purchased and managed for a tiny fraction of that money.
We're also seeing massive dividends paying out with this architecture because we have self-hosted gitea, along with a local workstation for our agents to run in, and now our agents have all of the context without us relying on Github or ingress/egress fees at all.
[edited for formatting only]
And every time it's a nightmare. I'm just banging out a server for my experimental card game, not setting up an new financial institution. Everything looks as if I'm preparing to scale to infinity tomorrow, with a staff of a thousand and a budget backed by VCs.
Fortunately there's Netlify and similar, who put a gloss on it so that I don't have to boil the ocean. I figure that one of these days I might actually be forced to learn IAM and VPNs and God only knows what else. Meantime, every time I touch it my eyes bug out.
It's a ghost of its former self, but I'd probably still rather use Heroku today than being forced to use Lightsail even once again.
Lightsail is pretty competitive (price wise) with other providers. Been running s B2B app on it for a few years now - nothing much, just your basic crud app running on lightsail instance + lightsail db. Nice to have a "monthly" rate on each instead of the EC2 opaque (and "surprise!") pricing.
Its not my favourite, but its not terrible.
You could do this, but for the life of me I can't imagine why you do this over using a platform like DO, vultr, hetzner or any one of a hundred similar services that will give you a better developer experience for this kind of workflow, often at a fraction of the price
You build something, well enough that it can handle the traffic, and people come, and it does.
Welcome to the gaming industry
Their help page lists a bunch of 2FA app options, all of which run on phones, so it's understandable to think a phone is required. (I'm disappointed they don't list the app I use, which is Aegis Authenticator.)
But actually you can use any TOTP app, and they don't all need a phone. For example, macOS (desktop) has built-in TOTP 2FA as part of the password manager.
For 90% of the time when they're up.
Use AWS at your own risk, Paul Vixie is not there to save you.
I will bite the bullet and pay for RDS because it adds a lot of value - scalability, a reasonably optimized config, backups I don’t have to worry about.
But Elasticache is exploitatively priced with almost no value add.
It is slower, less optimized, less stable, and only supports one DB compared to a vanilla redis install with zero configuration.
There are some scalability improvements, but it’s extremely rare they’re even required because vanilla redis so wildly outperforms elasticache on a similar instance.
AWS doesn't add much in terms of APIs or polish. On the other hand, Redis/Valkey is one of the most simple services to self-host.
Otherwise, some things that are good about AWS are as under:
1. IAM is I think good, logical and granular enough.
2. Separation of compute and storage in EC2 is very good.
3. S3 is amazing.
4. SQS is heavily underrated.
5. RDS is expensive but too good. I do not know how to go about 1 TB+ database size with daily backups without RDS. Similar ZFS setup with file system snapshots is complicated.
Not good things about AWS:
1. Super expensive. About 10 times. With zero support.
2. Current geopolitical environment would suggest getting off AWS if you are not a US company. The fascist idiots at the helm of affairs have lower IQ than the big void's average temperature in outer space.
EDIT: Typo + Formatting
1. IAM and policies. I’m not convinced that anyone knows how IAM rules and policy rules interact. There’s a flow chart that appears to be incomplete. There is not obviously a complete enough spec that one could, say, write a test suite to confirm that the actual behavior follows the spec. LLMs, of course, don’t know either because the training data does not exist.
2. Utter nonsense pricing. The cost of listing an S3 bucket goes up by an order of magnitude if you set the default storage class to archive despite this having nothing whatsoever to do with the operation in question. (But GCS adds two orders of magnitude for the same offense.) Conclusion: NEVER EVER set your default storage class to an archive tier.
3. Boto. It’s an Unbelievable Piece Of Crap. It’s not a library at all — it’s a meta-library that generates itself at runtime because someone had fun doing that and because Python didn’t stop them. Python type checkers, of course, just give up. And Boto is, um, a community project that AWS claims not to care about. Which is, of course, why its maintainers refused to fix an interop bug with GCS (I fully documented the entire bug for them, and the fix would have been the removal of a bit of pointless code).
4. Egress pricing. And the way it multiplies if you use any advanced VPC features. Why on Earth is it cheaper to sent an object to S3 from my own machine than to send the same object to the same endpoint from within a different AWS region nearby?
5. Authentication. It’s so bad that they invented Identity Center to try to unsuck it. But if you use Identity Center you get logged out even while actively using the console, and you get a helpful link to the WRONG PLACE to sign back in. Because of course core AWS isn’t even aware that Identity Center exists.
I don’t even use AWS very much. I’m sure I would fall in love with more of it if I did.
What most people realize, that you don't have to go microservice or fragment your code to a billion little repos, you could take a standard webserver, and move it to lambda, as long as you don't expect requests to be able to share on-server state.
This is always the weird things in those rants. He's complaining that after 4 days his mails are offline.
Now I'm doing a mix of physical servers in rented rackspace, and rented servers - but even there I can have billing mixups where they deactivate servers for no good reason. And to get email working again the limiting factor would be the DNS TTL - new servers would be online somewhere else within hours of it going down. (And yes, I tested that just last year - one hoster threatened cutoff due to non-payment on a paid invoice, which prompted me to move the mail server just in case while getting this resolved).
That he is complaining about his email being down or that he trusted AWS at all with email?
(to clarify: DNS+domain at the same service can be OK, as long as you have nothing else there. As soon as you start having other stuff, keep the DNS there, but move the domain registration away. Depending on which domain make sure you have auth keys, access to the admin domain or whatever would enable moving the domain without registrar cooperation. In my hosting days I did my fair share of emergency transfers and infrastructure to help companies get their basics online again after a SNAFU - totally doable to have first mail coming in again within a working day)
Yeah, no that's not how it works with email. You have to build reputation for weeks or receivers throttle you.
For outgoing emails, reputation is a huge issue, but at the same time it’s also fairly trivial to set up a (different) 3rd-party (gmail, outlook, sendgrid, whatever) with previous reputation so you can get back communicating.
Executives will always prefer to transfer liability and responsibility to someplace else.
Who's calling the shots in an organization? Engineers or executives?
Well, besides for the fact that the author's got suspended for no reason, WorkMail is being shut down March 2027 anyway. I recommend checking out Purelymail for a budget, batteries included option. Another option is to run your own server but have it use something like AWS SES to send externally, avoiding the IP reputation issue.
their dashboards are trash & don't work - Google Cloud, AWS Console, Google Ads, Meta Ad manager
I won't even mention the hyped up LLM vendors.
but here we r - people being laid off due to A.I - money being funneled into Gigawatt datacenters
lol ok. I have ~50 lambdas running in my personal aws account. Some of them are webservers running behind an api gateway or using a lambda function url to expose them to the internet. Some are running on a schedule, some are triggered from s3 events. The cost to run these for me is less than the cost of the cheapest vps (my total requests per month stay under the free tier limit). There is also zero maintenance I need to do for these functions (ok, this year I did have to find-replace al2 to al.2023 in my terraform config). I don't have to worry about making sure the os is patched for the latest vulnerabilities. And I don't have to worry about the specific hardware my code is running on at any time. Doing maintenance for old projects sucks. It is great to have servers I deployed years ago continue to chug along without me needing to think about it.
Now, all of my lambdas are written in Go and I suspect if I was using one of the manged runtime libraries I would find the language upgrades to be quite annoying. Go also helps quite a lot with cold start times.
Then again maybe I have just drank the koolaid. In my quest to use lambdas for as much as I can as cheaply as I can, I made a library[0] to use sqlite on top of s3 (not just readonly). It uses the sqlite session extension plus s3 compare-and-swap to allow you to write updates safely to s3, even if you have concurrent writers.
I don't think this is a valid argument. Free-tier VPS do exist also.
On the other hand, if you don't trust unattended-upgrades [0], and prefer to spend time poking package manager manually (while at the same time considering that time an expense) - sure, that's a strong argument in favour of using lambda.
[0] https://ubuntu.com/server/docs/how-to/software/automatic-upd...
Am I the only one who remembers that VPSes and dedicated hosting services were a thing before AWS came around? Yes you had to pay for a month at a time and scaling wasn’t as instant, but it wasn’t like the only option before cloud computing was having to drive to the datacentre and install your own server.
The “in minutes” is doing a lot of the work in that sentence above.
I also used dedicated servers in the late ’90s (and they still offer great value today). But before AWS, provisioning new hardware typically took days, not minutes.
AWS changed that, and the rest of the industry eventually followed.
The virtualised server thing was not a AWS thing, the thing that was were their other services. For example instead of renting a virtual server and installing a database on it. You could rent the database; that was sort of a new thing that AWS made in to thing.
It was never cheaper what you paid for was a promise of fire and forget. You would no longer need to worry about any responsibility to update the server or the database cause the AWS crew took care of that.
VPSes and non-custom configs for dedicated servers were pretty instant as far as I know, I think the advantage of AWS was more that you could scale up and down much more easily since you weren’t locked down in a monthly contract, and that you could automate server provisioning through an API.
We had super bursty traffic, and had to go with Google Cloud (very early days! [0]) because you'd need to communicate with AWS and pre-warm the ELB capacity of your expected bursts.
We did a dead launch to 60 million customers (0 to 60 million, no organic growth phase) this way. I wouldn't want to do that on a VPS.
I miss the Media Temple days.
Innovation has ground to a halt of mostly just meh “hey us too” launches. Pricing and design patterns feel increasingly focused on locking you in. AWS folks tell me internally they talk a lot about making sure things are “sticky” with customers. The best engineering talent no longer wants to work there and it shows, especially in places like AI where AWS has just released wave after wave of discombobulated nonsense.
As a core “rent-a-server” concept with a few add on services there’s still a lot of utility, but AWS is gradually becoming a boring baseline utility with a ton of distracting half baked stuff jammed on top. Most companies I talk to are no longer focused on single cloud and increasingly are bringing a lot of workloads back on prem or in colos. Not everything, but for a lot of stuff that just makes more sense and is a heck of a lot cheaper.
The chips business in Annapurna is probably the most interesting thing and that plays to its strength of the boring low level infrastructure stuff. Nearly everything AWS tries to do beyond chips and rent-a-server plays is a hot mess.
AWS isn’t going away, but its future looks a lot less exciting and inspiring than the story that got us to this point.
Nobody wants to hear this, but as things stand, there's no escaping risk for vibe coders right now. Personally, I think AWS is still a good choice for the long run, but don't make the mistake of thinking current LLMs will actually be able to manage the environment on par with a decent infra engineer. That's one of their weaker areas right now. Good news is there are million managed service providers and AWS-competent humans still in existence. Also Premium Support is a good resource.
Whatever you do, make a lot of backups and store them on a different service somewhere. Then if you get to a situation where you need to do something with sensitive data, or need to raise money, engage with someone who can do a proper review.
DX is simple, integrations between the two, and the stack is well understood by the LLM.
Lovable uses supabase, and is surprisingly easy to eject from too; I've done the lovable to Vercel + supabase a couple of times, even managing to keep it syncing via the Git integration. You can get proper scalable infra and minimal vendor lock in whilst the vibe coder gets to play with the pretty.
I saw some 192 core instances on Vultr, but I haven't tried them yet. What are you doing with all them cores?
I often fantasized about spinning up hundreds of nodes for various projects that needed number crunching. Then realized "wait I can just rent one big box for an hour" haha. It's really cool that we can do that now.
The ancient forgotten art of Vertical Scaling.
AWS IAM is extremely well designed when you compare it with the spaghetti monster IAM systems of other clouds.
Every time I try the new cool thing supposed to replace these services on some other provider - I understand how mature and polished the AWS ones are.
With that said, the rest 90% of AWS services like WorkMail, Cognito, API Gateway, are absolute hot garbage which no good meaning AWS expert will touch with a 10 meter stick.
Agree, so is STS and SDKs generally just work. I don't miss on-prem companies with legacy Auth where you maintained 100 service accounts for everything with very careful password vaulting and credentials management policies. So much easier to use IAM policies.
>are absolute hot garbage
I kind of like Cognito but both Cognito and especially API Gateway are somewhat convoluted to configure. They seem to work fine once you have them setup right, tho.
Talking about hot garbage... Not a fan of Redshift and Lake Formation at all. We switched to Snowflake, saved money, got better performance, and had a simpler setup. Really there was nothing about Redshift that was better. We're billed through Marketplace so there's not even a consolidated billing upside.
Imo Redshift is a relic of the past and has failed to modernize.
AWS used to have a nifty tool called "policy analyzer" or something that monitored for permissions used by a role so you could scope it down. The other day I had the need for it and when I went to use it, found out they charge something like $9/resource. So I would pay $45/month for metadata monitoring on just 5 things? Nuts. If they knew how to build truly delightful products, they would make something like a role that starts with broad permissions and automatically scopes itself down after some point. And it would be free or at least really cheap.
DDB is hardly a database. The only reason I can think of to use it is for massive amounts of data whose schema and query patterns are guaranteed to almost never change, which is very rare. Need to sort data on a field? Then you have to create a 'secondary index', which is a copy of the table that they charge you for and that is not strongly consistent. Schema change? Good luck with that. And don't you dare ask to use a nice ORM library. But hey it's serverless.
Here's a good one: you stop an EC2 instance and its volume keeps running and you pay for that. If you detach the volume, you still pay. There is no way to 'archive' an instance. And the only way I found out about that was I got hit with a big bill for those volumes with the charge labeled 'EC2 - Other' lol. Not very 'customer obsessed' to me.
My gripes are clearly not important to them because this is old stuff. So all I can do is go somewhere else, which is fine with me
1. You need an "infinitely scalable" key/value store and have deep pockets[0]
2. you work at AWS and your deployment pipeline has so many stages and regions and fabrics that you can no longer even conceptualize what it means for there to be a "current version" of your software (the hell in which I live).
But for some awful reason it's sold as a general purpose "NoSQL Database." Pair that with the Pavlovian response developers have to the word "scale" and you've got an army of people using the worst possible tech for their usecase. Everyone eventually pairs DDB with Elastic whenever "Oh, wait, so we need to be able to query our data?" hits.
[0] And you ONLY need PK reads. Querying turns "infinite scale" into "infinite throttles."
The writing has been on the wall for a few years now, and this is particularly evident to those thar have worked at AWS: Amazon is in its day-2 era.
Amazon being in its day-2 era means that most of what has been written in the past twenty years about Amazon is bot valid anymore.
“Customer obsession” is literally their first leadership principle, and stellar support was their defining characteristic.
I've had some run-ins with poor AWS support and have even gotten bill credits/refunds when they offered incorrect advice. It really depends on the service but in general they're fairly good.
yep, that's the point.
it used to be stellar on its own, now it's only good when you compare it to other vendors.
They've always struggled to hire for those roles. The people who are best at Engineering Support also tend to be the people who move on to other roles after a year or two.
but unlike OP I just accepted this fate and moved away from aws :)
I’ve a couple of apps doing a few million a day. I am using Hetzner and before that used DigitalOcean. Mind you, for close to a decade.
People are unnecessarily complicating stuff, and these clouds can go very expensive very quickly.
Recently, I came across a company and they were spending $20k a month on GCP. I am like, are you kidding me, $20K for the kind of stuff you do??? It seems you do not understand how CPU, RAM and Disk work to plaster such "autoscaling hyper solutions" burning money in cloud.
I moved their stuff out of the GCP managed solution and ended up with a $200-400 per month bill. The CEO can still not believe how it's even possible.
I suggested them move to Dedicated servers but they didn't want it, they said they must show they are on Hyperscaling cloud.
OK fine, we'll stay in Hyperscaler but not use any of their service other than VMs.
They racked up a ton of bills by using cloud monitoring, Datastore, and autoscalers (with no proper tuning), Kubernetes.
I replaced all of it with Prometheus, Grafana, Loki, and most stuff from Datastore to Postgres and Mongo with replicas. I added Redis.
I implemented a custom scaler where you can scale off of app metrics, not by just using a random peg on CPU.
I implement hot data reload by packing the data updates in gzip file, uploading to GCS and pulling from autoscaled units. Moved the stuff to Spot VMs.
The complexity of stuff in cloud is high for nothing.
At a previous bigger company, getting procurement to sign up to a new provider requires writing a business case, justifying the spend and then getting multiple competing quotes and speaking to their sales teams. Signing up to a new service takes _months_ even for $10/mo as they’ll negotiate for bulk discounts and the best possible terms for something that will literally cost less per year than one of meetings they hold to discuss the “value”. Meanwhile on AWS I can click a button in the marketplace and it gets thrown in the AWS account which is pre approved spending.
We use it because we don’t want to deal with slow procurement process. It kills all the momentum.
Watched one company end up with a $250k AWS bill when their credits expired (which they could not pay).
As an example, they needed a lot of proxy servers. Instead of just using a proxy service, there was a fleet of ec2 instances.
This is the main reason; and it applies to developers (they need cloud buzzwords on their resume), it applies to managers (who in turn hire only those with said buzzwords) and it applies to company execs/CTOs who can brag about the complex (self-inflicted) problems their company is solving at the next cloud provider conference, so they can justify yet another VC round.
Run this for over a decade, and you'll end up in a situation where an entire generation of "engineers" is no longer capable of configuring a Linux box to serve some basic webapp and will make up whatever reasons to avoid even attempting to do so.
I agree that it's overcomplicated. Although having the self-service portal also for assigning IPs is useful. But most of it seems overkill. Although, being able to detach storage from VMs and such is also quite flexible. But still.
Our 64 core spot instances on windows were taking 8-10x longer than our developer machines with the same core count, and there was a bunch of engineering went into the scaling, queue management, etc. if we’d just had a single bare metal machine from hetzner we could have saved money _and_ reduced our iteration times.
I can stand something up on AWS in a couple hours and be fairly confident it will run reliably (assuming their service offering is actually decent--some suck)
We test backups and they never fail. Metrics and logs always work.
>People are unnecessarily complicating stuff, and these clouds can go very expensive very quickly.
I don't think that's the cloud vendors fault. They make it easy to stand up new services so people get overly enthusiastic and create convoluted architectures. Have Postgres but need full text search? OpenSearch is just a few clicks (well hopefully IaC config..) away, let's use that! When you're building yourself and need to setup the stack, instrument, monitor, configure backups the cost is high enough where you say "hey, maybe pg fts is fine for now"
> burning money in cloud
I suspect there's two reasons why this happens.
One is just the disassociation with opex that seems ever present in the VC model. The other is that many startups settle in on a ops solution before hiring ops and the cost of switching isn't that attractive until they're faced with a dwindling runway and a down round.
Sounds expensive
Those tend to be tricky hires on the small end since you tend to want jack-of-all-trades who either demands a premium salary or doesn't exist.
When you have 10 software engineers, having 1 dedicate 10-20% of their time is cheaper than hiring 1-2 FTEs that aren't writing code.
If you can get (and trust they do it right) developers to do AWS or Kubernetes, you should be able to trust them to do conventional Linux sysadmin on a bunch of dedicated boxes.
By the time I joined, 18 months after development had started, a giant, complex, hideously tentacled software beast had been built that used every possible AWS service that the massive offshore team of developers could find to use.
It should have been built on a single Linux box by a single senior developer with Python and Postgres or nodejs or Ruby or whatever.
They went out of business after not too long and I couldn't help wondering if things might have been different if they hadn't spent a fortune building a giant money making machine for AWS, instead of making a web application on a Linux box.
Every AWS project I have worked on has had some significant work put into programming AWS instead of writing business functionality.
To be fair, if they had a AWS Solution Architect involved they heavily push you down this road and if they manage to get in management's ear they'll push the idea that server-less AWS features is vastly cheaper.
If you're only responding to a handful of requests that's true, but once things ramp up you get "nickel and dimed" for everything: API Gateway requests, lambda execution time, DynamoDB read/write units, CloudWatch logs, outgoing data, step function transitions, S3 requests.
I understand all those services cost money and they shouldn't be free, but I question if paying all those micro-transactions is worse then paying for your own VMs, especially once your customers complain about the cold starts and you think you can fix it with "lambda warming"
You moved something from a single datastore to three different database technologies? I don't know your domain, but that sure doesn't sound like a complexity reduction.
what's bad about graphana? it's simply used for some alerts and monitoring, i've used it for really long time and it has never failed me not even once.
it's much simpler to query postgres or mongo compared to duplicating data dozens of times on datastore.
You removed all of their logging and all of their redundancy and reliability and replaced it with shitters that will all explode if the small providers one data centre goes down.
And if someone penetrates this mega server, they’ll be able to wipe all your logs or tamper with them, to hide the attack.
If your storage servers go down, everything they have is gone. And these providers don’t offer the finest hardware. How do you know all of those drives aren’t from the same batch? They will be, because they’re a bulk buyer with a single data centre.
they'll never need it, a misconfiguration on those service ends up costing several grands.
>If your storage servers go down, everything they have is gone
It’s just logs for an app server, not some banking critical info that will cause a panic if lost. Most of what they are using for logging is for finding some errors, not for mission-critical things which must not be lost.
Because it's explicitly something you can request when doing your server order from your vendor. In this particular case several years ago, Nutanix did good.
Perfect explanation - no notes
I don't think I remember anything so over-engineered and confusing in recent times (probably SELinux now that I think of it).
And I understand - we kinda need the complexity for what they intend to do but they do need a Come To Jesus moment here to make the Insane Asylum Machine make a bit more sense for mortals
2nd most annoying thing? Boto3 lib, where conventions don't matter and Pythonic is just a suggestion and the thing works more like a REST wrapper than anything else over a not-great API (please why tell me there's an S3 API and an S3Obj API)
If anyone can clone any SaaS, then there will be millions of SaaS that offer all the features you need.
How will you choose?
AWS and Microsoft (and all the big clouds) make it easy for their customers to get hacked, and Mythos makes it more likely the cadence will only intensify.
But, if I vibe code a hosting service which is pure rust and doesn't use any external libraries and never open sources my code, my attack surface is much smaller and I only have three customers anyway.
Hackers are lazy and will go for the pond where the most fish live. AWS will always have a lot of marks and a lot of holes.
AWS will be expensive because you are paying the tax they have to add to fend off the hordes. It'll be an intelligent choice to avoid working with Rome and find a little village in Bergen.
AWS CLI??? Holy guacamole, what a mess. AWS CLI looks what is now the digital identification to get the basics done.
While GCP CLI is like "sure, here"!
You're also putting your business at risk with Google randomly banning accounts and not providing timely appeals. [1]
Hey good lookin'
Lambda is incredibly simple to use, it just runs a function for you.
Not sure how you could burn so much with dynamodb. It’s serverless and incredibly cheap. Must have been doing something insane like a huge dataset where you scan through it over and over.
Being salty that Gary couldn’t sell enough of his paid service and AWS is competing with it isn’t a meaningful complaint. I want something in AWS, not on Gary’s servers.