Why Rewind Said Hello to AWS Session Manager and Goodbye to SSH

Custom Tooling

Establishing a session manager shell is straightforward using the AWS cli:

aws ssm start-session --target "i-01234567abcdefg"

For an SSH tunnel, the syntax is:

aws ssm start-session --target "i-01234567abcdefg"
  --document-name AWS-StartPortForwardingSession
  --parameters '{"portNumber":["22"],"localPortNumber":["9999"]}'

At Rewind, we use Elastic Beanstalk for most of our services. As such, we are big fans of the eb cli. Switching back to using the instance ID was a little less friendly so we combined a couple of small tools together to make session manager easier to use. One of these is open source and the other is in a private repo with some Rewind specific parts to it.

aws-connect

aws-connect is a small CLI wrapper around the AWS cli that allows users to specify connection endpoints by instance Name tags rather than instance id. If I have a series of EC2 instances with a Name tag of web-server-tier, I can execute:

aws-connect -n web-server-tier

aws-connect then establishes a session manager shell connection to the first instance found with Name=web-server-tier. I can add the -a tunnel flag to create an SSH tunnel instead, or if I want a specific instance, I can add a -s flag to be given a list of instances with matching tags.

We also have another small wrapper around aws-connect called rewind-connect which understands Rewind infrastructure and prompts the user in platform-specific terms they are familiar with. This provides another way to zero in on a specific instance id before itself calling aws-connect. We distribute this internally with a private homebrew tap.

Synchronizing SSH keys for Elastic Beanstalk

One of the benefits to session manager is no longer being required to manage SSH keys. That being said, SSH tunnels do require keys and we’d rather not use the shared key provided by Elastic Beanstalk. How do we deal with this? There was a fantastic StackOverflow solution posted by Sam Maleyek. I made some tweaks to it and converted it into a full Elastic Beanstalk extension. Let’s look at the extension and what it does (this would saved to $project_root/.ebextensions/sync_ssh_keys.config or something similar)

files:
  "/home/ec2-user/get-ssh-keys.sh":
    mode: "000755"
    content: |
        #!/bin/bash# Only users in this group will have their keys sync'ed to instances
        iam_group_allowed_users=beanstalk-ssh-usersauth_file=/home/ec2-user/.ssh/authorized_keys
        tempfile=/home/ec2-user/.ssh/authorized_keys.tmp.$$logMsg()
        {
          INITPID=$$
          PROG="get-ssh-keys"
          logger -t ${PROG}[$INITPID] $1
          echo $1
        }# Delay the start so we don't get a storm of instances crushing IAM
        sleep_duration=$(( (RANDOM % 10)  + 1 ))
        logMsg "Delaying startup by ${sleep_duration} seconds"
        sleep ${sleep_duration}s# Save off the "root" beanstalk SSH key if we have not already done so
        if [ ! -e "/home/ec2-user/.ssh/root_key" ]; then
          logMsg "Saving beanstalk root key to /home/ec2-user/.ssh/root_key"
          head -1 "/home/ec2-user/.ssh/authorized_keys" > "/home/ec2-user/.ssh/root_key"
        fi# Set the perms now or SSH will fail
        touch "${tempfile}"
        chmod 600 "${tempfile}"
        chown ec2-user:ec2-user "${tempfile}"# Always start off with the "root" key
        logMsg "Initializing ${tempfile} with beanstalk root SSH key"
        cat "/home/ec2-user/.ssh/root_key" > "${tempfile}"allowed_users=$(aws iam get-group
                        --group-name "${iam_group_allowed_users}"
                        --query 'Users[].UserName'
                        --output text)for iam_user in ${allowed_users}
        do
          logMsg "Retrieving SSH keys for user ${iam_user}"ssh_key_ids=$(aws iam list-ssh-public-keys
                        --user-name "${iam_user}"
                        --query 'SSHPublicKeys[?Status==`Active`].SSHPublicKeyId'
                        --output text)if [ -n "${ssh_key_ids}" ]; then
            for ssh_key_id in ${ssh_key_ids}
            do
              logMsg "Retrieving public key for ${iam_user} with ID ${ssh_key_id}"ssh_key=$(aws iam get-ssh-public-key
                        --encoding SSH --user-name "${iam_user}"
                        --ssh-public-key-id "${ssh_key_id}"
                        --query 'SSHPublicKey.SSHPublicKeyBody'
                        --output text)echo "${ssh_key}" >> "${tempfile}"
            done
          else
            logMsg "No active SSH keys found for user ${iam_user}"
          fi
        done# Drop in the new authorized_keys file
        logMsg "Replacing ${auth_file} with ${tempfile}"
        mv "${tempfile}" "${auth_file}""/tmp/get-ssh-keys":
    mode: "000644"
    owner: root
    group: root
    content: |
      */10 * * * * root /home/ec2-user/get-ssh-keys.sh > /dev/null 2>&1container_commands:
  01update_cron:
    command: "mv -f /tmp/get-ssh-keys /etc/cron.d"
  02postinit_hook:
    command: "cp -f /home/ec2-user/get-ssh-keys.sh /opt/elasticbeanstalk/hooks/postinit/12_get-ssh-keys.sh"

Let’s have a look at the interesting parts…

iam_group_allowed_users=beanstalk-ssh-users

We will only synchronize SSH keys for users who are a member of this IAM group.

head -1 "/home/ec2-user/.ssh/authorized_keys" > "/home/ec2-user/.ssh/root_key"

Always save the root Elastic Beanstalk SSH key to fallback on should we somehow lock out all users.

ssh_key_ids=$(aws iam list-ssh-public-keys
                        --user-name "${iam_user}"
                        --query 'SSHPublicKeys[?Status==`Active`].SSHPublicKeyId'
                        --output text)

This is the crux of the solution. Here, we get all active public SSH key IDs for a specific user.

IAM allows uploading SSH public keys to an IAM user for use with CodeCommit. But nothing says we can only use them with CodeCommit! These keys can all be managed by the users themselves and rotated like IAM keys. Perfect!

ssh_key=$(aws iam get-ssh-public-key
                        --encoding SSH --user-name "${iam_user}"
                        --ssh-public-key-id "${ssh_key_id}"
                        --query 'SSHPublicKey.SSHPublicKeyBody'
                        --output text)

Next, we can get the actual public key and write it to the authorized_keys file on the instance

container_commands:
  01update_cron:
    command: "mv -f /tmp/get-ssh-keys /etc/cron.d"
  02postinit_hook:
    command: "cp -f /home/ec2-user/get-ssh-keys.sh /opt/elasticbeanstalk/hooks/postinit/12_get-ssh-keys.sh"

Finally, we will synchronize the keys every 10 minutes in a cron on the instance. We also add to a Beanstalk postinit hook in order for the keys to be synchronized right away at boot time, rather than potentially waiting for the first run of the cron.

Wrapping up

Having worked in tech for 25 years, two principles have really stuck with me in regards to security:

• Defence in Depth: The more layers you can implement to secure your stuff, the better. Session manager has allowed us to remove any SSH access and shared keys. Combined with full session logging and IAM, we have a much more secure access model for the times when we need shell access to an instance
• Security is not convenient: A security officer told me this over 20 years ago. I’ve generally found this to hold true over the course of my career, however, AWS session manager is one of the rare exceptions. Session manager is both secure and convenient!