AWS Session Manager: less infrastructure, more features
github.comAre they basically trying to emulate GCP’s OS Login (https://cloud.google.com/compute/docs/instances/managing-ins...) feature here? We’ve been using that for a while, and it’s been a big relief.
It seems like a re-implementation of cloud shell for Azure (https://docs.microsoft.com/en-us/azure/cloud-shell/overview). It also uses the browser and native IAM, but you can use the native AD integration and JIT permissions.
But it seems no one like Azure these days, but they have some nifty features.
os login is probably a little closer to ec2 instance connect because you still need ssh inbound access right? whereas aws provides a bastion here
There is a proxy called IAP [1] which is used to create SSH tunnel over HTTPS to instances without public IP.
[1] https://cloud.google.com/compute/docs/instances/connecting-a...
You're right in a sense, but there's no aws-managed bastion. Session manager communicates with your instance via an outbound-created websocket connection. Inputs and outputs are piped through it.
yea, i was trying to keep things simplified, but it has to proxy through something behind the vpc endpoint. could also say its not technically ssh
Though you can actually get ssh through ssm: https://docs.aws.amazon.com/systems-manager/latest/userguide...
Yeah, this is the same deal. Session Manager will log your sessions which is pretty cool.
I'd love to learn how you're using Session Manager or what other features/integrations you'd like to see us explore. Also if the terraform module packaging is useful. There are additional Session Manager features like port forwarding that I plan to write about soon.
Can you write one about port forwarding? Specifically, I would like to understand how various web interfaces on EMR cluster can be accessed through Sessions Manager. (Ganglia, Spark history server, etc.)
We'd love to use Session Manager, but we're running into the same issue mentioned here:
"Tunnel created using SSM only allows single connection to destination port" - https://forums.aws.amazon.com/thread.jspa?threadID=314882&ts...
This has been sitting open in the support forums unanswered for over two months :/
Cool, will do!
I have bunch of questions that stop me from deploying SSM into real world production scenarios.
1) Is logging for access from CLI finally supported?
2) Can I setup which shell is used?
3) Are logs readable when I switch to something else than sh?
4) Is U2F supported (awscli question)
Once all of these are fixed, then it can be possible to claim that SSM solves these issues. Otherwise it’s nothing more than for adhoc usage.
1. does it work with hardware tokens ?
we have some regulatory requirements that require us to use hardware tokens for 2FA access to servers.
2. what about SSH tunnels ?
It does work with hardware tokens, IF you get your AWS IAM credentials using a hardware token. If you're using AWS IAM users then here are instructions: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti...
If you're doing a federated login with Okta or another provider, you need to set up the hardware MFA there.
There is SSH tunneling support as well, will add an update on that soon.
Our organization recently looked into AWS Session Manager for tunneling but couldn't find documentation on how to make it work for our usecase. We were trying to tunnel into our VPC in order to be able to connect to an Amazon DocumentDB cluster. We don't have any EC2 instances which seems to be the only thing Session Manager has support for. Despite the callouts that Session Manager replaces bastion servers, that didn't seem to be the case for us. Did we miss something in our research?
Last I checked the "tunneling" only works to redirect traffic to a different port on the same SSM managed instance. The tunnel cannot be established with another box in the same VPC. So I don't think you can call it tunneling until they add that feature. Here's the GitHub issue where they discuss the limitation and a workaround: https://github.com/aws/amazon-ssm-agent/issues/208
I never see mention of Windows with Session Manager. I have a mixed infrastructure with a number of Windows (IIS) app servers running various things.
We currently connect via SSH to a bastionhost, then tunnel from there to various systems, which allows connecting to SSH (linux instances), RDP (Windows), or basically any other network services like Redis or a database. I ended up writing some scripts to automate all this, so as long as you have the right certificates and IAM permissions, you can connect with a single command -- for Windows instances, it even retrieves the randomized password from the EC2 API. The end result is for any EC2 instances you're instantly popped into a shell/RDP session without having to enter credentials.
I'd love to replace this with something better (eg Session Manager), but I've not seen how to do this for RDP, and haven't had the time to go experimenting on my own to see if it's even possible. If I can't 100% replace the bastionhosts, having two entirely different connection methods doesn't solve anything (and in fact makes it worse, because it's harder to use).
Have a google, I was using SSM for remote access to Windows instances, specifically headless instances.
Be careful with SSM in general. The documentation suggests adding the AmazonEC2RoleforSSM policy to the role of the EC2 instances you want to access via Session Manager. This role grants read/write to all S3 buckets in your account (amongst other things). See this article for better steps and unavoidable risky things: https://cloudonaut.io/aws-ssm-is-a-trojan-horse-fix-it-now/
There are so many AWS managed policies that provide access far beyond what one might suspect given the policy name and description. Implementing least privilege with IAM can be really difficult in any moderately complex environment.
And close to impossible in highly complex environments. We're looking to adopt using AWS accounts to achieve a few objectives but one being reducing the blast radius of potential breaches and outages.
Good call to watch out for this stuff. The examples in the repo we set up use the AmazonSSMManagedInstanceCore managed policy, which does not grant any S3 permissions, just various ssm, ssmmessages, and ec2messages permissions.
> The documentation suggests adding the AmazonEC2RoleforSSM policy to the role of the EC2 instances
Which documentation do you mean? The article mentions the policy AmazonSSMManagedInstanceCore, which is the same as what's mentioned in the SSM setup guide:
https://docs.aws.amazon.com/systems-manager/latest/userguide...
Thanks for clarifying, I didn’t recheck since we rolled out SSM in mid-2019 and then scrambled when we realised we’d granted account wide S3 permissions. The article I linked to also has a recommended minimal IAM policy for Run Command and SSM. I’ll update my comment to mention this.
It looks like the docs were update in June 2019 (https://github.com/awsdocs/aws-systems-manager-user-guide/co...)
AmazonSSMManagedInstanceCore is still too much access, it has unscoped ssm:GetParameter! I hope you weren't trying to protect any secrets in ParameterStore!
Doesn’t that still fail if you don’t also have the corresponding decrypt KMS permission?
This has not been true for a while. The IAM policy also indicates it is deprecated in the description.
Though I always suggest to read what managed profiles are doing. Many of them are very permissive
I wrote a follow up post to this on SSH tunneling: https://news.ycombinator.com/item?id=22665037
Does anyone know how this works with other utils that use SSH protocol, like rsync? What about tunneling other services to or from a local host? I'd love to have fewer hosts to maintain and a smaller network/attack surface, but we use SSH for more than just gaining commandline access to our instances.
The author states:
>"No more bastion hosts required! Session Manager uses AWS APIs to communicate with your instances, so you can remove the administrative burden of maintaining bastion hosts."
Does this presume the EC2 instances have a public IP or is there a way this would also work with EC2 instances on private subnets?
We've been using the Session Manager with instances in private subnets without issue, works like a charm.
Is there anything special that needs to be configured to get this to work on private subnets?
Currently I have an EKS cluster accessible only on private subnets. It would be wonderful to to be able to access this without OpenVPN in the mix.
The instances establish an outbound connection to the API SSM API, so as long as they can hit that, Session Manager will work.
Connectivity from a private subnet to the AWS API could be (a) NAT gateway (b) HTTP proxy (c) PrivateLink VPC endpoint.
It's great for managing active SSH sessions, but not so much for the other purpose for bastions: fine-grained network access control+routing. It would be cool if they made a more specific version of this just for network traffic without the SSH component.
FWIW, the project is open source, so you could build a modified agent for your purposes and inject it via cloud-init or your favorite config management tool: https://github.com/aws/amazon-ssm-agent
Hey, what about the Werbot solution - werbot.com? Now it is very relevant.
IAM is easy to mess up.
Would be interesting to lock down the session manager agent (if possible) so that the only way to privileged access is through sudo-like priv esc that uses 2fa.
You can do this but it depends on your setup as to how. If you have AWS IAM users (not federated), then you can use MFA conditions in your policies as documented here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti...
For federation you need to rely on the config in whatever your identity provider is, like Okta.
Its fairly trivial to lock down AWS via a require MFA policy
I'm talking about on the host, so if you mess up your IAM policy there is still an authorization layer on the host to get privileged access.
As far as I know, SSH over SSM doesn't do anything regarding user management. It just establishes an SSH connection. Management of users on the host, authorized SSH keys, etc. is totally out of scope for SSM.
So if you already have access control setup on your host, then SSM doesn't do anything to undermine it. If you don't have it, you'll still need to add it.
Does anyone know of it works with Ansible ? Thanks!
This is awesome! How can I install the the agent if I'm not using Amazon Linux?
Amazon installs it on some other AMIs (notably, Ubuntu 16.04 and 18.04), but for other OSes, install instructions are here: https://docs.aws.amazon.com/systems-manager/latest/userguide...
Note that you won't get this feature at the free tier for non-EC2 instances.