21 More AWS Services They Should Cancel
justingarrison.com+1 on NAT Gateway. For those unaware, you need to setup a NAT gateway for your tools inside of a VPC to access the internet. I forget the pricing, but it's way more expensive than it should be and it's a huge pain to setup. This is a service that is annoyingly expensive for hobbyists/indie-devs/people just playing around, but a rounding error for AWS's "real" customers. Just build it into VPC (a checkbox that says "I would like to be able to access the internet from my code in the VPC") and make it free or at least have upfront pricing.
> you need to setup a NAT gateway for your tools inside of a VPC to access the internet
You do, if your stuff is in a private subnet. If you are just "playing around" however, you have options:
a) Spin up your resources in a public subnet, give then a public IP(be very careful about your security group rules if you do this)
b) Create your own NAT gateway EC2 instance(can be way less expensive than a NAT GW as tiny instance sizes can forward a lot of traffic). It's almost trivial to do. Disable source/dest check, enable ipv4 forward, configure routes.
c) IPV6 :) Depending on what your destination is (+ an egress only IGW)
I wouldn't recommend either (a) or (b) for a large production environment, but small deployments will do fine. You can't escape network egress charges though.
You could do A but in addition to the security issues now you have to pay for public IPv4s on AWS too so if you have a significant number of services that are private but need internet access it is still cheaper than NAT gateway but just barely.
I've done B before for dev environments and it works well. For production there is a large list to make it high availability.
Which brings up one of the travesties of NAT Gateway is if you have a dev (or more) and staging and you want it to match prod you're all the sudden stuck with a paying for multiple NAT gateways.
> if you have a significant number of services that are private but need internet access it is still cheaper than NAT gateway but just barely.
Also depends on the volume of traffic we are talking about. NAT GW is $0.045/h even if doing nothing, plus $0.045/GB, plus egress. IP is $0.005 without any extra costs other than the standard egress.
> For production there is a large list to make it high availability.
Yes! Which is why I wouldn't do it in production unless your org and team structure can deal with it. The problem is solvable technically(and that's how we used to do things before the service existed) but the people problem is trickier - this kind of infrastructure runs a high chance of getting neglected and mostly forgotten until it causes an outage. Outages (often due to instance 'maintenance') caused us to migrate away from using our own NAT. If they cause you to lose money, or spend a bunch of engineer hours, there goes your savings.
AWS NAT Gateway is pretty reliable in comparison and you mostly forget it exists. The problem is just cost - you pay per hour, and you per for egress on top of the usual egress charges. So AWS is double dipping there.
I wish AWS had the same underlying VM tech as Google. GCP can migrate systems to another hypervisor without start/stop and without even dropping network connections. Unless the underlying hypervisor dies with no warning, having the ability to keep your connections up would avoid some people getting paged, even if HA kicks in.
> NAT GW is $0.045/h even if doing nothing, plus $0.045/GB, plus egress. IP is $0.005 without any extra costs other than the standard
That's only 10 servers. I sometimes forget they charge per GB too. That particular charge rarely affects me but if your private services need a lot of data that can certainly add up.
To expand on that, additionally, if you are running your own NAT you need to have one instance per AZ or you end up with cross-subnet transfer costs. So that's at least one cost that you save with NAT gateway (though moot if you run all your services in the public subnets)
AWS policy on NAT Gateways is so stupid that people came up with a d) option - alterNAT[0] that is basically b) but turns on the real NAT GW if b) fails giving you the best of both worlds: lower cost and better reliability than a NAT instance.
https://www.lastweekinaws.com/blog/an-alternat-future-we-now...
AWS just isn’t for hobbyists. You have to deal with the complexities of it because the real target customers want and need these things. There are plenty of other cloud services appropriate to your scale. It’s frustrating because you’re using the wrong tool for the job.
It's not the complexity (IMO), it's the cost. A hobbyist can easily set up NAT gateway but very often the NAT gateway is the most expensive part of the entire cloud bill. So the hobbyist is left with paying it or exposing their server to the public internet. It is very expensive for what should be something that is a built in part of VPCs.
Heck, even if you're not a hobbyist, I've worked with companies that have dev environments that mirror production (except smaller instance sizes) and now all the sudden you have a ton of NAT gateways eating money for providing a basic networking service.
It is but you need to consider cost first, not walk in with your existing assumptions about how to build stuff.
To be fair, large corporations probably should develop that mentality rather than shovelling vast amounts of cash into the problem and hoping it will go away one day (Hint: it doesn't).
One of my services is still running on Elastic Beanstalk. There's a lot of pros, but the cons are starting to build up, especially since it's relatively easy to port stuff to Elastic Container Service. I'm in the process of doing that now.
EB was great at the beginning, but as the number of connections to other AWS and third party providers increased, it has become incredibly clunky to tweak the build files. What's even worse is that when something does go wrong -- which it inevitably will -- it's extremely hard and slow to debug exactly what went wrong.
Using it with celery has also been rough.
All of that being said I'm ambivalent about having EB on this list. I think it's a good product that has languished partially due to three reasons:
1. AWS having too many overlapping tools (AppRunner, Lightsail, ECS, etc.) 2. AWS haven't added or prioritized as many new features on EB for years 3. The devOps tooling is much more mature these days.
EB has languished for 1 reason. The team that built/maintained EB was reorged to build AppRunner (as EB v2) and they never had enough cycles to maintain both and weren't allowed to deprecate v1.
You'll have to pry CDK out of my cold, dead hands.
What programming language do you use with CDK?
A year ago I tried CDK using python and all commands ran incredibly slow. I'm curious if that was fixed, or if it's a problem with other languages too.
Did you start with CloudFormation and move onto CDK, or did you start with CDK?
The reason I ask is to determine whether you try to reconcile what's going on in CDK vs what's going on in CloudFormation.
Years ago I started with CloudFormation, then moved to Terraform because CloudFormation lacked a lot of features back then. About a year ago I tried CDK. My impression was that CDK makes it very difficult to know what's actually happening under the hood. It's bad enough to have to understand CloudFormation, but the CDK adds more complexity over CloudFormation with the intent of being "magic" enough that people don't need to know how it works.
And maybe that's fine. Perhaps people can be productive without knowing how something works. But that drives me insane. When technology is magic and something goes wrong then it's impossible to fix (fixes require understanding what went wrong). Also magic makes it difficult to predict the effect when you make changes.
CDK does two things:
- It provides a library in your favorite language to map objects -> CFN YAML
- It provides a command line to deploy that CFN YAML + any needed resources, like Lambda payloads. This is done with `cdk bootstrap` [0]
Mapping objects to YAML is straightforward -- every language has some way to serialize arbitrary objects to YAML. CDK provides level 1 constructs [1] with the allowed CFN types. I use TypeScript, so these types are _super_ helpful, especially with Intellisense. I'm not sure how helpful they are if you're using Python.
Once you have those level 1 constructs, you can build abstractions on top of them. That's what the more complicated L2/L3 constructs are.
Because the end result is CFN YAML, you do still need to understand CFN. CDK just makes the authoring experience significantly more pleasant by eliminating the need to write CFN YAML. You still need to know how to deploy CFN YAML.
[0]: https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html
[1]: https://docs.aws.amazon.com/cdk/v2/guide/constructs.html#con...
I think TypeScript has a much better experience with CDK.
I love that NAT gateway and Cloudformation+CDK+Proton are on this list. I've spent the last 3 years building with these tools, and these are some of the most frustrating things in our stack. Our NAT gateway costs are gross. We pay to have the ability to send traffic, but we don't use it enough for that price. Meanwhile, CDK hid the awkwardness of Cloudformation from us for awhile, now we're too tangled to easily leave it.
Most of these sure.
Workmail is a weird choice, I mean its just fine as an email system so why cut it? Plus migrating email hosts is kinda a pain. I use it for my personal/professional email and, I never interact with Workmail directly since its on my phone.
Lightsail, it sounds like the primary reason is that it won't make money on its own but I figured that was never it's purpose. It always seemed like Lightsail existed to get people into AWS for smaller projects.
Cloudformation I strongly disagree with. I use it for all infrastructure and having used terraform in the past I much prefer it. My only real problem with CF is that it is not required that all new features are part of CF when they launch. That fixes CF but doesn't need a replacement to make it work. Throw in some UI improvements, but no reason to throw it out.
I don’t claim that CF is a brilliant product but over the years I’ve come to appreciate that for typical workloads I can get it done with virtually no extra cost and without third party components and configuration. I’ve been trying out CDK and honestly I think it’s an iterative improvement that CF sorely needed.
When I saw Nat Gateway and Cloudformation, I feel like OP should switch to DigitalOcean or Linode for the simpler stacks. AWS is not for custom solutions. Without Cloudformation, how do you manage your stacks? Terraform is third party and now getting complicated with the license.
Edit Nevermind it’s just me. Should have checked my extensions first.
FYI to the author, there’s a curious bug with using the back button on iPhone/safari.
If from this thread I click the link to go to your page, and before doing anything hit the “back” button, this works fine.
If I go to your page, and then scroll to the bottom, and then hit back (like I did after reading the article), safari seems unable to do so, staying on the page and appearing to get caught in a loop with a growing list of history links to the same page. I think something about the jump links is confusing safari. I do have various extensions installed in safari, so if no one else has this issue it may just be me.
This does not happen in the iPhone/orion browser, which function fine.
Thank you very much for the note. I use hugo to render the site and don't have any specific setting for back button behavior. I'll check it out.
Shoot, looks like just a me problem after disabling some extensions. Currently narrowing down which one is misbehaving. Sorry for the false alarm, should have tried this first.
Edit looks like an issue with “Hush Nag Blocker” (https://apps.apple.com/us/app/hush-nag-blocker/id1544743900) which was originally installed to deal with the cookie nag spam epidemic. I’ll have to look into alternatives.
You just saved me 1-2 hours trying to replicate the problem tonight. You earned an internet cookie Thank you for being a good internet citizen.
How about evolving CloudFormation and CDK instead of trashing it and starting over.
The CF web interface does not do a great job of showing the hierarchy and relationships between resources.
And working around circular dependency issues in CDK can be a real time sink.
> The CF web interface does not do a great job
At anything. To say nothing of the godawful expression language in yaml, so CF is bad UX at pretty much every level. Even ops, the thing it’s made for, is excruciating: I just want to see deployment progress per-resource so I can see what’s stuck, and maybe in some distant joyous future, why.
Any amount of evolving would require a lot of breaking changes to rearchitect and not be bogged down by keeping compatibility. I think they should make a v2 and sunset v1 and not keep any old compatibility.
There are a lot of pitfalls in CF. A lot of lessons you have to learn the hard way.
I’ve had the same experience with TF and Ansible with the added bonus of having to pay out the nose for the privilege.
Just signed up to say that I'm an AWS Certified Solution Architect and I've used none of them in production and don't even know what most of them are.
And no we don't use CDK or CloudFormation!
That's because AWS is creating very few new "core" products. Everything seems to be a hosted version of software someone else created. Or perhaps a product to hop on the latest tech fad (Blockchain, LLMs, etc.)
Yeah true. It's all ElasticWeForkedSomeOssStuff and the likes. Each fork of which is 10x more expensive than running it on a flat EC2 or EKS cluster.
> and App Runner so unsuccessful,
Is App Runner not doing well? I've been using it and it seems... well not great, but I am surprised that it's not at least trending upwards.
Everything trends upwards. Even the services they killed in the past year I’m sure were getting new customers. But Amazon isn’t interested and doesn’t have capacity to support hundreds of services that don’t make a lot of money.
I knew someone a while back who worshipped the ground AWS stepped on, and thought CloudFormation and CDK was the best thing since sliced bread.
I honestly couldn’t see where they were coming from. CF is awful to work with, even more awful when you have to recover from a failed deploy or rollback, and hacking declarative concepts into Typescript for CDK is just a maintenance disaster in the waiting.
I don’t know why you would choose any of those over Terraform or equivalent declarative IaC tools unless you are a die hard AWS fanatic or simply had the misfortune of inheriting the stack.
CDK lets you write Python instead of configuration files. It’s a good idea, but hobbled by CF being a steaming pile of garbage underneath it. TF was building a CDK competitor a while back but progress was slow on it.
I noticed TF has JS examples in the docs now. I haven’t looked but if it has the same problem as CDK, where you have to shoehorn declarative concepts into an imperative language, you still end up having to write weird code to work around the fact that you don’t actually know the value of resources until they’ve been created or modified.
Now if there was a prolog version? I’ll have some of that.
You still need to run a JS environment even if your own CDK code is in Python. At that point you may as well just use JS if you're going to have to put up with its drawbacks anyway.
How good is AWS at a) communicating impending deprecations and b) providing lead time to migrate off deprecated services?
In the last year they've deprecated 14+ services/features and they've been really bad. They will email existing customers but they won't announce it publicly to avoid the bad press. Documentation pages are updated quietly with banners or removed.
Recently, I attended an hour long meetup from a high-level AWS employee about CI pipelines using CodeCommit. Of several possible deprecation announcements, the latest of those was dated the day of the talk. (!!)
In all fairness to the speaker, except for having one of the most prominent icons across his slide deck deprecated in real time, it would have been a pretty decent talk. He even made an effort to promote the GitHub integrations as a path forward, and provide some guidance on current tooling. It was clear CodeCommit wasn't the path of most momentum, even if the degree was unclear.
What do you propose to replace CloudFormation with, especially since you’ve gone scorched earth on the related products?
I would be fine if they built a new tool with 2024 IaC experience and control. But I think trying to evolve CFN into a new thing would take far too long and have a lot of edge cases that they should just start over and stop trying to paper over it with CDK, Proton, ACK, etc.
CDK seems like it could support multiple backends, and in fact there’s cdk8s already (having never used cdk8s, I can’t comment more on it). The CF data model seems fine enough though, so I think overall CF just needs a lot of love. Not holding my breath though.
I use cdk8s to manage my homelab [0]. It's a great experience.
Embrace OpenTofu?
What services would you get rid of?
A huge portion of AWS services are really annoying in my opinion to grassroots developers and platform managers because they are basically executive demoware.
You know the kind where the salesman comes in and in front of the CIO builds some whiz-bang demo and like 20 minutes and has a CEO asking why it takes a month or more to do equivalent stuff by real it workers.
And sometimes it's worse to customize the out-of-the-box solution than just creating your own solution. For some AWS products, it's pure pain to get it up and running in the configuration you require. There are edge cases and bugs to worry about.
That whiz-bang demo? Maybe that's the only functionality that works right. Maybe it's all using default values that won't pass your internal security and compliance policies.
And lets not forget the pain of integrating something new with existing systems. It's easy to show a demo of something that doesn't integrate with existing systems, and just show a slide or two of what things it integrates with.
Doesn't Amazon/AWS use this very heavily? Whenever we have to get on a call with AWS engineers it's through Chime.
As a product, it seems fine. I'm not entirely sure it's an area AWS really needed to have a competitor, but now that they do, /shrug
At every org I've had to interface at a deep level with AWS, they've used our Zoom. n=1
If you get rid of chime you also get rid of slack huddles (built on Chime). Like many AWS services the backend of Chime is good, the UX/UI is terrible.
TIL, interesting design decision on Slack's part.
It was a business decision because slack's old /call command was built on a startup they aquired. I don't know the details but I'm sure Amazon gets deep discounts on slack and slack get's the same for chime.
https://slack.com/blog/news/slack-aws-drive-development-agil...
Agree with all of those