Curated AWS SSM Scripts

We’ve written in a previous article about Rewind’s use of AWS SSM Session Manager and associated IAM policies to allow shell and SSH tunnel access to AWS resources.  We also open sourced a small tool we created to make use of SSM easier called aws-connect. But what if we want to allow users to only run a subset of approved commands rather than full shell access on an EC2 instance?  We have a solution for that!

Systems Manager Documents

AWS Systems manager has a feature called Run Commands which allow you to run scripts or commands on EC2 instances that have the SSM agent installed.  The commands are based on documents where a document is usually a script with some optional parameters.  The great thing about these SSM documents is that access can be controlled with IAM policies.

So what does an SSM document look like?  It’s a YAML or JSON file that looks something like this:

schemaVersion: "2.2"
description: "Hello World"
parameters:
parameters:
type: "String"
description: "Some text to print"
default: "none"

mainSteps:
-
action: "aws:runShellScript"
name: "runShellScript"
inputs:
workingDirectory: "{{.}}"
runCommand:
- "echo 'hello world'"
- "echo {{ parameters }}"

This simple example just takes in some parameters and prints them along with the string “hello world”.  SSM documents can be invoked via the AWS console by selecting the document and choosing run command or it can be invoked using the CLI or API.  Here’s how we’d invoke this on an instance using the CLI:

aws ssm send-command
--document-name "admin-hello-world"
--document-version "4"
--targets '[{"Key":"InstanceIds","Values":["i-123456789"]}]'
--parameters '{"parameters":["param1 param2"]}'
--timeout-seconds 600
--max-concurrency "50"
--max-errors "0"
--region us-east-1

OK, so we understand the basics of Systems Manager documents, what’s next?

SSM Documents from Github

While the above sample document shows a script embedded in an SSM document, it is possible to have an SSM document reference code in Github directly (either public or private repos).  This is incredibly handy because we can now have all the features Github gives us for source control (including Pull Requests for peer review).  We now have curated SSM scripts!

How does an SSM document look when it’s pulling code from Github?  The big piece is the addition of an action that is based on aws:downloadContent

action: "aws:downloadContent"
name: "downloadContent"
inputs:
sourceType: "GitHub"
sourceInfo: "{"owner":"rewindio", "repository":"my_private_repo","getOptions" : "branch:main","path" :"scripts/read-only/", "tokenInfo":"{{ ssm-secure:{{githubTokenLocation}} }}"}"
destinationPath: "{{ workingDirectory }}"

The important parameter required here is the ssm-secure parameter.  This is the path to an AWS parameter store secure string parameter.  This parameter must contain a Github Personal Access Token (PAT) that has rights to clone the specified repo. In the example above, the PAT must have clone permissions to the rewindio/my_private_repo repository.

Let’s look at this in the context of a full document.

schemaVersion: "2.2"
description: "Run Github script"
parameters:
githubTokenLocation:
type: "String"
description: "Location in ssm param store of your github token"
default: "none"
workingDirectory:
type: "String"
default: ""
description: "(Optional) The path where the content will be downloaded and executed
from on your instance."
maxChars: 4096
executionTimeout:
description: "(Optional) The time in seconds for a command to complete before
it is considered to have failed. Default is 3600 (1 hour). Maximum is 28800
(8 hours)."
type: "String"
default: "3600"
allowedPattern: "([1-9][0-9]{0,3})|(1[0-9]{1,4})|(2[0-7][0-9]{1,3})|(28[0-7][0-9]{1,2})|(28800)"
mainSteps:
- action: "aws:downloadContent"
name: "downloadContent"
inputs:
sourceType: "GitHub"
sourceInfo: "{"owner":"rewindio", "repository":"my_private_repo","getOptions" : "branch:main","path" :"scripts/read-only/", "tokenInfo":"{{ ssm-secure:{{githubTokenLocation}} }}"}"
destinationPath: "{{ workingDirectory }}"
- precondition:
StringEquals:
- "platformType"
- "Linux"
action: "aws:runShellScript"
name: "runShellScript"
inputs:
runCommand:
- chmod a+x my_script.sh
- ./my_script.sh
workingDirectory: "{{ workingDirectory }}"
timeoutSeconds: "{{ executionTimeout }}"

In this example, the path scripts/read-only from the private github repo rewindio/my_private_repo is cloned.  The actual run command then runs a single script from here called my_script.sh.

SSM Document Permissions

So we have these SSM documents, how can we assign permissions so that only particular users can execute particular documents on specific instances?  Here’s a sample IAM policy that we’ll step through:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SessionManagerSendCommandSpecificDocs",
"Effect": "Allow",
"Action": "ssm:SendCommand",
"Resource": "arn:aws:ssm:*:*:document/scripts-read-only-*",
"Condition": {
"BoolIfExists": {
"ssm:SessionDocumentAccessCheck": "true"
}
}
},
{
"Sid": "SessionManagerTomyGreatProductInstances",
"Effect": "Allow",
"Action": [
"ssm:StartSession",
"ssm:SendCommand"
],
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"StringLike": {
"ssm:resourceTag/product": "myGreatProduct"
}
}
}
]
}

The first block allows access to SSM documents beginning with the prefix scripts-read-only-.  Users (or roles) with this policy block will only be able to run documents that are named beginning with scripts-read-only-.

The second block allows running commands only on instances that have a tag called product with a value of myGreatProduct.

Using combinations of these blocks assigned to different IAM users/groups/roles, you can restrict access to specific SSM documents and EC2 instances.  In our case, we have a couple of user “profiles” we use depending on what access is needed. This is controlled at the IAM Group level and the appropriate IAM policy assigned for the subset of curated scripts we want to allow access to

Auto-Generating and Publishing SSM Documents

We have most of the pieces of this solution in place now. But how can we automatically translate code in a private Github repo to SSM documents automatically?  Enter Github actions.

We’ve created a publicly available Github action which automatically creates SSM documents for code added to a Github repo.  This action will generate the correct YAML for the SSM document and fill in the correct script name to execute.  The beauty of this is now script authors only need to focus on writing their scripts – they don’t need to know all the syntax or nuances of the SSM document schema.

How do we use this Github action?  Here’s a sample workflow (this goes in your .github/workflows folder in the Github repo containing your scripts)

name: synchronizeSSM

on:
  push:
    branches:
      - main

jobs:
  release-staging:
    strategy:
      matrix:
        publish-region: [us-east-1, eu-west-1]

    name: ${{ matrix.publish-region }}-staging
    runs-on: ubuntu-latest
    env:
      PUBLISH_REGIONS: ${{ matrix.publish-region }}

    steps:
    - uses: actions/checkout@v2
      with:
        fetch-depth: 0

    - name: get files changed
      uses: lots0logs/gh-action-get-changed-files@2.1.4
      id: get_files_changed
      with:
        token: ${{ secrets.GITHUB_TOKEN }}

    - name: generate ssm yaml file
      id: generate_ssm_yaml
      uses: rewindio/github-action-generate-ssm-documents@main
      env:
        FILE_LIST: ${{ steps.get_files_changed.outputs.added }}
        PREFIX_FILTER: scripts
        DEBUG: True
        AWS_ACCESS_KEY_ID: ${{ secrets.STAGING_AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.STAGING_AWS_SECRET_ACCESS_KEY }}
        REPO_NAME: my_private_repo
        REPO_OWNER: rewindio

In this example, SSM documents will be created for any files checked into the repo and pushed to the us-east-1 and eu-west-1 regions.  Let’s dig into some of the pieces of this workflow:

jobs:
release-staging:
strategy:
matrix:
publish-region: [us-east-1, eu-west-1]

name: ${{ matrix.publish-region }}-staging
runs-on: ubuntu-latest
env:
PUBLISH_REGIONS: ${{ matrix.publish-region }}

This uses the matrix strategy for jobs.  Github actions will dynamically create a new job for each entry in the publish-region list, naming it using the region name and setting an environment variable in each job with the name of the region.

- name: get files changed
uses: lots0logs/gh-action-get-changed-files@2.1.4
id: get_files_changed
with:
token: ${{ secrets.GITHUB_TOKEN }}

This uses a handy public action to get the list of changed files in the current change to the repo.  We want to generate SSM documents for each file changed.

- name: generate ssm yaml file
id: generate_ssm_yaml
uses: rewindio/github-action-generate-ssm-documents@main
env:
FILE_LIST: ${{ steps.get_files_changed.outputs.added }}
PREFIX_FILTER: scripts
DEBUG: True
AWS_ACCESS_KEY_ID: ${{ secrets.STAGING_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.STAGING_AWS_SECRET_ACCESS_KEY }}
REPO_NAME: my_private_repo
REPO_OWNER: rewindio

Here’s the special sauce.  This is our Github action that generates the SSM document(s) for each changed file.  In this case, it uses PREFIX_FILTER to only look at changed files under the scripts folder.  The AWS keys referenced must be stored as Github encrypted secrets for the repo and the corresponding AWS IAM user must have enough permissions to be able to upload SSM documents (ideally no more than this)

So putting this all together in a simple “what does this do” statement:

Whenever files are added to this github repo under the scripts folder, a corresponding SSM document will be created in the us-east-1 and eu-west-1 regions.

Invoking Documents using aws-connect

The final piece of our solution is an easy way to execute these scripts. Given we already have our aws-connect utility, it made sense to extend this. Here’s an example of it in action:

aws-connect -x i-23323ere3423
-r us-east-1
-a document
-d my-ssm-document
-p my_aws_profile
-w 'param1 "param 2"'
-g /devops/github_token
-c ssm-cloudwatch-logs

We’re asking to run the my-ssm-document ssm document on instance i-23323ere3423, passing parameters param1 and param 2 to the document.  The gitub PAT is located in the /devops/github_token SSM parameter store parameter.

Pulling it all together then, we’ve got a solution where IAM users can be given access to run a subset of curated scripts against a set of tagged instances.  We’ve used this successfully to limit access to more powerful scripts to only our senior engineers and it’s a great example once again of using the incredibly versatile SSM service.

Like putting these kinds of solutions together?  Check our careers page for all the engineering roles we’re hiring today.