AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely
aws.amazon.comFirst reaction: Holy crap! They finally turned ParameterStore into a proper product!
Second reaction: Holy shit that's expensive [for what it does].
ParameterStore is free (minus the KMS component). The only value-add is secret rotation and that's not something that most of the time makes sense to use. [Edit: I'm not advocating for no rotation; see replies]
Edit: Had more time to think about it. Someone enlighten me: What's the difference between writing a rotation lambda for this new product, vs. writing a rotation lambda for ParameterStore that you then cron? The pricing really doesn't make sense.
My biggest criticism of Parameter Store is actually that it's free. Let me explain:
Because it's free they limit the requests per seconds you can make to the Parameter Store. That's especially noticeable when doing requests for all parameters of a given path, as the limit is way lower there than for requesting (a bunch of) individual parameters. In the past that caused serious problems for us when using Parameter Store for AWS Lambda functions during a deploy of new versions of functions, as suddenly there was a spike in the number of requests to Parameter Store as all AWS Lambda containers got replaced.
They of course set such limits because it's free, so I'd gladly pay for getting increased limits.
Oh yeah, I agree - and I have no beef with the per-request pricing. It's the per-secret pricing that's weird. If I understand this announcement correctly, you're still paying for KMS separately as well, so that's not where the price spike comes from...
Oh, yeah, we just ran into this recently when rolling-out parameter store. For anyone else having this problem, the trick was to use the GetParametersByPath API method (https://docs.aws.amazon.com/systems-manager/latest/APIRefere...) instead of the more flexible DescribeParameters (which has stupidly low, undocumented rate limits).
At the time we experienced the problems GetParametersByPath had way lower limits than GetParameters (~an order of magnitude)! That's the problem we ran into. That was already back at the end of last year and the SSM team promised to improve the situation, but so far at least no publicly available news states something about it.
> The only value-add is secret rotation and that's not something that most of the time makes sense to use.
From a security perspective, you should be rotating secrets somewhere between annually and every 90 days, depending on your business/security/compliance requirements and the nature of the data secured by the secret.
You're not necessarily your own source of secret (and even when you are, you don't necessarily have the option to use AWS-sourced rotation).
In other words, yes, you should be rotating what you can rotate, but this doesn't always help. Furthermore, it doesn't justify the pricing. It would make sense if this were, say, "Hey, you can now auto-rotate SSM-PS secrets for an extra $0.40/secret".
Right now it just seems weird.
[Edit: I just saw the custom rotation bit of the article. Cool; but if you're at the point of setting up lambdas for the rotation, you might as well cron a lambda on top of ParameterStore...]
Does it matter? How many secrets do you have where this is even close to your ec2/storage costs?
It depends on the secret and the degree to which the secret is exposed. SSH creds should get rotated constantly; a one-hour SSH login cred is a significant exposure. But an API secret that is kept in Parameter Store and not exposed to developers doesn't really benefit from rotation every 3 months in proportion to the amount of mechanism required to do that.
What are you assuming about exposure? If a SSH key lives on a well-secured workstation or bastion host (and you ideally don't agent-forward it to insecure hosts), rotating that key once per hour doesn't seem a top priority to me? E.g. a sudo password is (lower-impact, but) more likely to get exposed to compromised hosts?
("Well-secured workstation" is arguably an oxymoron, of course...)
Wait, you keep private SSH keys on bastion hosts?
It's much better practice to use the bastion as a proxy to the other hosts. This is easily achieved using the ProxyCommand option of OpenSSH.
No agent forwarding, no secrets kept in random hosts.
No, keeping SSH keys on bastion hosts was the silliest reasonable thing I could think of.
Do you know a good guide with standard practices for setting up and securing bastion hosts (preferably on aws)?
If you don’t require interactive sessions, consider using AWS SSM run command [1] instead. You install the agent on the instances, with commands sent from the client through the AWS control plane (with IAM and SSM documents for access control and CloudTrail logs of all commands issued).
I’m currently deploying it in an enterprise for ~5k users, and it works surprisingly well for providing the ability to run arbitrary commands on instances without ssh access.
[1] https://docs.aws.amazon.com/systems-manager/latest/userguide...
...a one-hour SSH login cred is a significant exposure.
Does this refer to an SSH password or e.g. a 4096-bit RSA key, or possibly something completely different? If this refers to an RSA key then I'm quite confused.
How are you managing your SSH keys to make <1HR rotation feasible?
With an SSH CA.
Is there one of those you’d recommend?
Check out Hashicorp Vault (https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-cert...) for a general product (has lots of the features of AWS Secrets Manager as well), or Netflix BLESS (https://github.com/Netflix/bless) for a very specific tool for just this.
What's the rationale behind rotating? I've never really understood this. I'm asking this from a fundamental point-of-view: I realize there are policies and standards that require rotation, but I want to know why those policies say that in the first place.
If you have a secret, it's either (potentially) compromised and should be changed immediately, or it's not compromised and thus doesn't need to be changed.
If you're not sure if it's compromised or not, you have an entirely different problem that isn't solved by rotating. Put another way, rotating every 90 days is the equivalent of saying "it's okay if this secret is compromised, because no one can do any damage with it if they only have under 90 days" -- and I can't imagine any situation in which that would be true or acceptable.
You seem to be thinking of a detected compromise, which may have compromised secret X, and your response is to change secret X right now. And I think that's the correct response.
But in another sense, all your secrets are potentially compromised, all the time. That is, there are compromises that you don't detect, or don't detect right away. They happen. What should you do? Well, you change the secrets every 90 days, even if you don't (yet) know that they have been compromised, to minimize the damage of a compromise that you don't know about.
But, you may say, by that logic shouldn't we change them every day? Every hour? But the other tradeoff is with annoying your users, wasting peoples' time on changing secrets, wasting time as they have to try to remember today's secret, and all the associated hassle. It's a compromise between better security and better usability.
Is 90 days the right compromise? I don't know.
If you're talking about secrets used by users (which isn't what AWS Secrets Manager is used for), that is specifically covered by NIST 800-63B [1] which recommends:
* Do not impose other composition rules (e.g. mixtures of different character types) on memorized secrets.
* Do not require that memorized secrets be changed arbitrarily (e.g., periodically) unless there is a user request or evidence of authenticator compromise. (See Section 5.1.1 for additional information).
There's research [2] backing the idea that these policies actually lower security because of the way users act in response.
Systems don't have these problems, so rotating every few seconds doesn't seem to be a big deal. However, if the secret was compromised and you don't know how, you have to assume that the new secret can also be compromised (via the same mechanism), and at the same time there's no reason to assume that an attacker can't make use of it before the next rotation. Thus the rotation is completely ineffective as a means against this unknown attack, so why bother? The energy would be better spent eliminating potential for unknown, undetected attacks.
[1] https://pages.nist.gov/800-63-3/sp800-63b.html#-1021-memoriz...
[2] https://www.ftc.gov/news-events/blogs/techftc/2016/03/time-r...
> The energy would be better spent eliminating potential for unknown, undetected attacks.
Think both/and, not either/or. Yes, work on eliminating unknown, undetected attacks. But you're never going to be perfect at it, so also work on minimizing the damage from undetected attacks. If you're not dealing with human-remembered secrets, sure, go down to rotating every few seconds. By doing so, you make the compromise harder to exploit. Even though you don't make it impossible to exploit, "harder" is still worth doing.
Got a link for rotation time frame recommendations because I have seen them all over the board. I understand the basic one that you want old employees to have no knowledge of current credentials but I have worked with security rotation requirements for minutely, hourly, daily, monthly, and yearly and I've never really seen a reasoning for the difference between them.
The pricing is a bit surprising, but I am sure that if you tried to build the full solution you would end up needing a few more pieces than just the rotation lambda and cron.
Would have been nice if there was a 5 secret free tier though. I feel like the sticker shock will dissuade small users when everybody really should be doing secret rotation.
After Amazon EC2 Systems Manager Parameter Store got some much needed love last year (adding KMS encryption and versioning), I wrote a small utility for populating environment variables using a specific Parameter Store prefix: https://github.com/nlindblad/confidential
It can either write the environment variables out to a Docker compatible environment file or it can be used as a wrapper to run any executable/script with the environment pre-populated. The README contains various examples of those use cases.
The primary motivation was to have proper secrets management for my hobby projects that didn't rely on Ansible Vault, configuration files in S3 or having to use the AWS SDK in each project.
Aren't all your secrets available as environment variables on your instances? If someone has shell access or gets the environment access pragmatically, all your secrets are compromised.
True, but if someone gets shell access as your application user, there is nothing stopping them from impersonating your application or reading any configuration files (including AWS credentials) stored on the file system.
If you are using IAM roles, a HTTP call to http://169.254.169.254/latest/meta-data/iam/security-credent... would give any attacker with shell access the credentials needed to iterate through any secrets in the Parameter Store that the instance has been given access to.
For shell access as another (non-root) user that is not running your application, it is just as safe to rely on environment variables as having configuration files with proper file permissions: https://security.stackexchange.com/questions/14000/environme...
Wrt rotation theres also a cultural component of it. Systems grow in complexity and importance. Forcing periodic rotation, With corresponding automagic client support, ensures thats its even possible when you need it. Compare STS tokens to IAM credentials.
What's the backend? If it's HSM, that would explain the steep price (sort of).
Would be very surprised if it's not backed by AWS HSM. About 2 years ago I believe AWS KMS-->HSM was leveraging SafeNet's Luna HSM.
The SafeNet product is now called "CloudHSM Classic". The current CloudHSM is documented to use the Cavium NITROX HSM, which in the PCI HSM format is partitionable into "virtual HSMs". This looks pretty much designed for cloud HSM providers.
Secrets Manager is very likely backed by KMS which appears to use a KMS-specific HSM module (judging by https://csrc.nist.gov/csrc/media/projects/cryptographic-modu...)
How many secrets do you have? Even with thousands this doesn't seem expensive, and if you have 1000's you're probably mature enough to pay a bit.
To be clear, this would be peanuts to my company. But it's super expensive for what it is especially because they have a free service that does the same thing right next to it.
Totally had the same reaction. The first thing I thought of is "wait doesn't ParamStore already do this?"
>The only value-add is secret rotation and that's not something that most of the time makes sense to use.
This attitude is why company after company keeps leaking customer data.
This hasty drawing of conclusions from a post that wasn't suggesting you shouldn't rotate secrets is why populist after populist keeps getting elected.
Is it really a "hasty drawing of conclusions" when it's based on a simple reading of what was written?
[secret rotation,] that's not something that most of the time makes sense to use.
I just had a quick glance at my own app. The majority of the secrets in it are generated outside of AWS and (unless I'm misunderstanding the announcement) can't be rotated by this service without custom logic to rotate them; at which point, as I said in a separate post, I might as well write that custom logic on top of ParameterStore and cron it.
This looks like an AWS equivalent of the Amazon-internal secret management tool called Odin. Which is very nice because Odin was pretty much universally loved from what I saw.
There's still a bootstrap issue with AWS Secrets Manager - you have to set up enough tooling to be able to call the API.
The killer-app of odin imho was/is the on-machine http server that let all manner of applications very easily get credentials.
You could just do something like
and boom your shell script or whatever was very easily using secrets.MY_API_KEY="$(curl http://localhost:5000?key=my-api)"The fact that Amazon systems bootstrapped EC2 instances meant they could easily enable this org-wide. The odin back-end then owned all the lifecycles around those secrets including what kinds of hosts they would go to, if/how they would rotate, etc etc. It has its annoyances including the fact you have to use http to localhost which can get saturated if you're not doing things right, but overall it really made secrets-management a non-issue.
That is actually pretty scary from a security perspective because of SSRF attacks. Basically if I can get your service to make an HTTP request on my behalf then I can get at your secrets.
Suppose you have a service that fetches snippets from user pasted links (like Slack does). All I have to do is paste a link of "http://localhost:5000?key=my-api" and your server would return a snippet containing the secret.
Thats why Google Cloud and others require a special header to be set. Hopefully Odin does the same? https://cloud.google.com/compute/docs/storing-retrieving-met...
This is already an issue on EC2.
https://ionize.com.au/stealing-amazon-ec2-keys-via-xss-vulne...
This is exactly how things already work on EC2.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-role...
TL;DR;
will get you the credentials for s3 access if you define a role named s3access and grant that role the associated permissions.curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
After leaving Amazon you really go on to understand how good the internal tools are at Amazon and how they probably have the best tools in the industry.
I had the exact opposite reaction after leaving Amazon. Brazil is a slow, bloated pile of crap that regularly blocked fixes, made it impossible to develop in a non-standard language, and regularly resulted in 100+ dev build blockages due to versioning issues.
Our team was abandoning Apollo for MAWS, so our deployments were bridged between the two, resulting in another twisted pile of incomprehensible script-glue.
Pipelines was ok (!)
Everything on public EC2 for managing deployments (CodeDeploy and CloudFormation in particular) were so broken and bug-ridden and non-performant that we couldn't reliably deploy production code with them.
I prefer open source solutions in almost every way. I actually can't think of a single problem that the Amazon internal stack solves that isn't better solved in the open-source world. Which isn't surprising, given the constraints involved.
Brazil + Apollo + Pipelines. I left (and later boomeranged) and was surprised that the outside world hadn't solved these problems nearly as well. Don't get me wrong, Brazil and Apollo probably are ready for a rethink and rewrite, but they do a great job.
if you went to another bloated enterprise company, that wouldn't be surprising. If you went to a high-profile consulting company and had that experience, I'd be shocked.
What would you change about them?
Hard to say (especially in as public forum like this).
Brazil's dependency model is, I think, very much worth rethinking. Just try to use it for NodeJS. Apollo is fine but slow and dated and needs love. Pipelines works for specific software models and not others (ever seen a pipeline with 500 envs?). Great design for building custom approval steps though (long ago I wrote a now-popular one).
Except SIM.
The Lotus Notes of Amazon
+1 Hahaha! This made my day.
As a former Googler, I know a lot of Googlers that feel exactly the same way. It would be interesting for someone who has deeply experienced both to weigh in.
Brazil/Apollo needs huge renovation though, especially when Containers are getting so much more popular.
Odin works well, it's just a hassle to share credentials with non-developers who don't have a dev desktop and therefore can't access Odin. It's an issue I've encountered several times recently with no clear solution, it seems.
Odin works really well but but only because it integrates so tightly with with the rest of the internal tooling. Without Apollo (another internal tool) to manage and orchestrate the deployment of packages that need Odin's secrets, it's a lot clumsier to use.
Surprised at all the folks openly discussing internal tooling.
There are a specific set of internal things at Amazon that public discussion of would be detrimental to your career. Saying what you think of Odin isn't quite the same as mentioning the release date of the next Kindle.
Of course, and I don't think anyone would or should get reprimanded for it. That said, I still think it's poor form and should be avoided unless absolutely necessary.
I miss having ODIN.
For those of us living in the service-development stone ages, is the idea that a secret-manager service replaces any number of ad-hoc local secret-storage and configuration mechanisms with a single robust mechanism that takes only a single root credential to retrieve all the individual secrets that your service needs?
You do still have to figure out a way to securely provide the root credential to your service so that it can fetch the secrets from the secret manager, correct? Otherwise this would be magic of a kind I think is impossible.
If my questions aren't too far off in the weeds, then this service sounds like a personal password manager but for a service rather than a person, though I'm sure AWS's service has finer-grained controls than just the all-or-nothing master passphrase. Similar risks apply: an attacker obtaining the master passphrase is a major issue, losing the master passphrase is devastating (though recoverable here because you probably didn't lose your personal AWS login credentials), and unavailability of the password database is catastrophic. But the usability benefits of having everything in one secure place, behind a service managed by experts, should outweigh those risks.
I have more questions about the credential-rotation feature, but this is enough for now.
The big missing piece is roles. No service uses a root access key directly. Instead, there's a webserver role with access to a relevant secrets group but no access to data warehouse secrets, for example.
Access keys can be provisioned and downloaded straight onto the box from the service. Sure, a compromise is bad, but only exposes the secrets that would be available on the pwned box regardless.
OK, so "root" wasn't really the right term. I get an X credential so I can be an X, and nobody needs to worry that I also got enough to be any part of a Y. Thanks.
This sort of model also fits nicely with the AWS ecosystem. EC2 instances (virtual machines) can be given an IAM Role when they boot-up. An IAM Role is essentially an automatically generated access key which is unique to that EC2 instance, and has pre-determined permissions.
So in other words – a unique key is generated every time a virtual machine is created. It's fully automated, never shared between instances, and never needs to be handled manually. That key will give the virtual machine permission to access other AWS services, in this case the AWS Secrets Manager.
So as long as you're using EC2 instances, you won't need to worry about securely passing a 'master password' to your VMs in order for them to access secrets.
I see. So there actually is a bit of magic involved.
I exist, therefore I can.
Yep - they have similar functionality for task roles (aka docker containers on ECS) and Lambdas as well.
It's a non issue to grant access to a particular group of secrets to a particular server role, assuming the requesting server has an IAM instance profile attached to it. The IAM Policy just describes what secrets the role can access, and then the server simply makes the api call to get the secret.
Having your personal AWS keys compromised should also be a non-issue because of MFA. My keys that I keep in OSX Keychain grant zero permissions, except the ability to assume a more privileged role, but that requires MFA.
FWIW, we're using Parameter Store for secrets and it works great. Given that I just finished that set up just weeks ago, I'm in no rush to jump on the Secrets Manager wagon based on what I'm seeing.
The only problem with both services is the 4k character limit. We have some big cert chains and such that exceed that, so we had to build the tools such that they could put/get a chunked secret. Wasn't too bad though.
At $0.40 per secret per month and $0.05 per 10,000 requests this is much more expensive than the practically free SSM Parameter Store product, even if you factor in the auto-rotating bits.
It'd be cheaper than running Vault with a backing Consul cluster which also provides rotation and other features.
There is a point where Vault is more cost effective, but I believe it'd require a ton of requests and secrets to justify min 6 machines of at least t2.micro that also need to managed and secured.
Depends on how many secrets you're storing.
You can also back Vault with something other than Consul. You can back it with DynamoDB, which would be much cheaper than managing your own Consul cluster. You can even back it with S3, which would be dirt cheap (cost of the vault instance + a few cents for storage).
I wasn't aware you could use Dynamo or S3, that's pretty interesting
It's definitely not cheaper than KMS and DynamoDB via Credstash, though.
What if you're already running a nomad/consul cluster? Is vault a particularly hard thing to implement/scale at that point?
Yea it that case then it's not that hard or extra cost. But I imagine most people aren't already running Nomad or Consul and can benefit from this. Lambdas, Wordpress, etc. can now get rotating secrets which is pretty nice possibility now with a lot less operational overhead.
...and that's without a FIPS 140-2 Level 3 backend for a HSM as well, right? (Secrets Manger has that I assume).
With Vault 1) you have to get Hashi Enterprise 2) Pay for a very pricey HSM.
No, I believe they just use KMS or GCM either which is backed by an HSM and it's recommended in their documentation.
Agree the pricing is way off, but at least you can use an object/table inside a secret to serve as a group of secrets. eg, not just key:"value", but key:{key:"value", key:"value", ...}
Nothing stops you from storing JSON in ParameterStore.
Yeah, I'm surprised they charge per secret. I guess it's a blip when you have big aws spend
Yeah, I don't think too many people have tens of thousands of secrets, generally. The addition of key rotation is a great benefit.
per 10,000 requests
Besides the secrets rotation how is this different from EC2 Parameter Store? I’m genuinely curious and will move away from parameter store if this provides some benefits.
Skimming the blog post, the main difference seems that it allows you to basically store a dict of key/value pairs for each secret. So for example, you could store all the user/pass/host/port for a DB connection as a secret. If I recall correctly, Parameter Store could only store a single SecureString for each secret.
ParameterStore lets you store String, StringList or SecureString. But there's no limit to SecureString.
A SecureString can be `postgres://admin:hunter2@localhost:5432/db`. It can also be `{"username": "admin", "password": "hunter2", "host": "localhost", "port": "5432"}`.
Sure you could do that if you don't mind adding in your own string->dict conversion. I wouldn't be surprised if the internal implementation is near-identical. Secrets Manager seems like a slightly more easy-to-use version of Parameter Store that's also visible, instead of hidden away inside Systems Manager.
It did take me a minute to actually find Parameter Store. There's so much stuff hidden away in the EC2 menus.
$$$
> $0.40/secret/month
WOWZER. I get having a managed solution is great, but you don't have to store many secrets before running your own Vault server makes sense.
At my work, we have a vault service, but it takes 3 servers for vault plus another 3 for consul and they are not very cheap servers (m4.larges). Our rough calculations, using a 3x3 vault/consul architecture, estimated you'd need over 1100-1300 secrets to make it worth implementing vault (not including development/maintenance cost which could be significant. and if you have multiple, independent environments, it gets worse.
The real issue for some of the people at my work is the vendor lock-in versus vault.
Consul only makes sense if you are already using it for something else. If just Vault, you can use DynamoDB (with full HA) or S3 (without HA support), same with Azure and Google Cloud (although without HA). There are other backends, like PostgreSQL.
If you're already deployed inside Kubernetes or similar, Vault is Just Another Pod that is running, with HA and everything. You might not even need to add to your resource pool that is already provisioned. As for your backend, S3 is a great choice if you can tolerate no-HA, or DynamoDB if not.
Point being: if you're using a modern stack, you can use modern tools. If you're stuck on a legacy stack, you use legacy tools, and that appears to be who AWS is building for nowadays. Legacy tools, value-based pricing instead of cost-based, charge a boatload, buzzword heaven, sell it to the enterprise, rinse repeat.
The HSM component is the only part here that appears to be worth the price. Azure's competing product is a LOT cheaper, except that their HSM-generated keys are more expensive. Which naturally means, Azure is cost-based pricing, whereas AWS is value-based pricing.
How many secrets do you think you'll have? 400? 4000? We're still within pennies range compared to doing this securely yourself.
FTFU - Valut service.... You need more than one server, it'll need to have 100% uptime too, backed up and available in multiple regions. No security team want to own this infrastructure.
Yeah, it's a minimum of 2 instances per region, and you should also run Consul or some other replicated backend, etc. Been there, done that; it gets complicated pretty quickly.
That being said, if the security traffic and/or amount of data is huge, Vault might be cheaper, even with the extra work. I guess it depends on the scenario.
Not to mention the Hardware Security Module(s) behind this are very expensive, especially when scaled out for redundancy and availability.
Kinda similar to what Hashicorp's Vault does for secret management but hosted.
There's really no comparison. You could use Vault to build a PKI. You couldn't say the same for a proprietary, vendor locked solution like this.
I believe that's what HC's Vault Enterprise is (hosted/SaaS): https://www.hashicorp.com/products/vault
I'm still trying to figure out how different this is from their "parameter store" offering in AWS Systems Manager. The main thing I guess is you get more control over key rotation.
Looks like the equivalent of Vault? Anyone can compare the two?
I am curious as to how it handles those race conditions where a connection is made with the older credentials just after the time the rds master key rotates, or a connection is made with the newer credentials just before the time the rds client key rotates. Short of using two credentials accounts ...
there’s a pretty nifty 4 stage rotation mechanism that solves most, but not all, of those race conditions. More info on it in the post and the docs.
Are we talking about these? createSecret setSecret testSecret finishSecret
These were mentioned around the custom rotation strategy, any pointers to docs describing how it is done if the secret is an RDS Secret?
As others have noted, the pricing is really the odd thing here. AWS seems to be moving to value based pricing rather than cost based pricing for some of its niche products.
I used to think that Cloudwatch metrics were very expensive at $.50 per custom metric per month, but this seems waaayyyy cheaper to store.
How is this different than KMS? “Key Management Service” is practically synonymous with the name of this new product, so how exactly do the two differ/interact?
A key difference is in kms you never know the actual secret (kms stores this on your behalf) . This is for use cases where you need the secret in application code, like db creds.
Looks like a cool product I noticed they dont include a revoke workflow though, the trickiest bit of key rotation
KMS is a shared HSM, managed encryption product.
AWS Secrets Manager is for storing credentials to products (e.g. DBs) and have processes for automatically rotating them on a regular basis.
KMS is the service which performs encryption where this stores the secrets using a specified key (provided or from KMS).
KMS provides an alternative to an HSM in the cloud where this service is alternative to running a Vault cluster with a backing Consul cluster.
Well, you can use KMS to do something similar, but without the auto-rotation.
1. Use a KMS key to encrypt a secrets file (obviously, never check this into source control)
2. Store the encrypted secrets file in an S3 bucket
3. Tie a new IAM role with kms:Decrypt and s3:GetObject policies for the relevant resources to your EC2 instance
4. On app start, get the KMS key and secrets file, decrypt, and set environment variables
In practice, rotating using this scheme just means creating a new KMS key, re-encrypting the file and pushing the updated copy to S3, and updating the IAM role's kms:Decrypt policy. It's not too bad unless you have a million services.
This seems like the easiest solution, and we've been pretty successful. We have sub-folders based on your environment and only machines have roles that can read production. We also realized you don't even have to encrypt and decrypt.. just turn on encryption for the S3 bucket, I believe it's the same thing.
It's not exactly the same thing. Encryption at rest on S3 is good, but security is about layers. If you only use that, then you are one mis-configured S3 bucket policy away from exposing your secrets. Bad bucket policies are one of AWS' biggest footguns- plenty of intelligent people have done it before, don't assume it's a mistake you'll never make. It's so easy because S3 is a service that has to make publicly-accessible storage easy to achieve.
KMS keys on the other hand, do not have that use case. If the secrets in your S3 bucket have been encrypted using a KMS key (or better yet, a data key derived from a KMS key) and your bucket is compromised, your secrets are safe as long as the attacker cannot ALSO get access to the KMS key. By all means, enable encryption at rest on S3, but don't make that your only defense.
I believe KMS is primarily for storing encryption keys (to replace an HSM). AWS Secrets Manager looks like it's much easier to integrate into credential best practices, like periodic rotation.
At $0.40 per secret it would have been nice if they had a 5 secret free tier.
That would get smaller users to start using it instead of parameter store and eventually realize the value of automated and audited secret rotation
Yeah I'm thinking about starting to use it for personal projects but don't want to bother with these 2 dollar bills. I'm pretty sure they can benefit from me playing with it first and then advocating for it at work
Does the Secret Rotation for the RDS-integrated credential store actually update the password for the user in the SQL database?
If so, thats pretty damn cool.
Oh, I misread it as a secret for Twitter (which, I guess, makes no sense, because tweets are generally public).. and I was wondering how the password got updated on the Twitter side. So I guess, naturally, secrets can only be rotated for AWS services that support it.
You can write the custom lambdas to rotate 3rd party creds.
Yes!
There is a lot of commentary about the use of vault as an alternative, number of secrets needed, etc. I think the inclusion of Secrets Manager is a great addition for AWS and will definitely help people get better control over their secrets, however, vault contains richer functionality than just secrets key/value storage.
It can provision users to backends like SSH, databases, cloud providers, and such. Use is audited, can be revoked, and has a TTL associated.
Additionally, vault contains a full "crypto-in-a-box" implementation that allows for sign/verify, hmac/verify, encrypt/decrypt, random number generation, and other functions.
So I applaud AWS for doing this and hope the will continue developing KMS/HSM/Parameter Store/Secret Store/??? in the future and innovating, but evaluating Secret Store vs. Vault simply on price may be a short sighted comparison.
Disclaimers: Employer is an AWS customer using vault
Compare this to the cost of Hashicorp Vault. You might be surprised at the difference.
$0.40 per secret per month is outrageous.
I was hoping to use this for RDS parameters (to get automatic rotation) and leave everything else in ParameterStore.
Unfortunately — although I'm sure it's built on-top of ParameterStore internally — I just checked and you can't see SecretsManager secrets in ParameterStore, so an application would need to read from both and merge them, or switch entirely to SecretsManager to take advantage of the automatic rotation.
This will be nice for systems and budgets which can't afford a consul+vault cluster to handle this for you.
"Not just for secrets" Yay, we can go and store all types of secrets!
But not with this price per secret...
Anyone know how this compares to Azure Key Vault?
Vault already did secrets. Nothing stops you from writing an azure function to handle rotation, but you gotta build it yourself. AWS has BUILT IN support for RDS.
However, the biggest delta is: I'm like 90% sure Azure key vault doesn't have fine-grained access policy per VAULT. That kind of stinks.. you need a vault per role ideally.
Au revoir Cyberark. I won't miss you.