Picture this...

It's 2 AM, most people are asleep, and your phone starts shaking like crazy!

Oh no! It looks like the production database has been deleted (I know, it should never happen just like that, but you never know!). No worries, though! We can restore it from a snapshot, but you'll need the necessary AWS permissions. Typically, database creation/modification/deletion should be handled through our Infrastructure as Code (IaC) Continuous Integration/Continuous Deployment (CI/CD) tools, like Atlantis for Terraform, which have the required AWS permissions.

As pipelines take on a more prominent role in the software development lifecycle in a DevOps model, the necessity for extensive human access to environments decreases. Human users should be granted minimal access necessary for their role, which is usually read-only access that does not allow any modifications or access to sensitive data. For experimentation which is typically hands-on and exploratory, teams should be granted access to sandbox environments which are isolated from system workloads.

~ AWS Well Architected Framework: Limit human access with just-in-time access

However, IaC with CI/CD requires both a pull request (PR) creator and a PR approver - always the two, like a master and an apprentice.

If you are the only person awake at night, making fixes through CI/CD becomes impossible (no one can approve a PR, <panic-mode-on>). So, what can be done in this situation?

If an incident occurs in the middle of the night and we lack the necessary permissions and cannot reach anyone for assistance, a break-glass policy comes into play. It’s like breaking the glass to access a fire extinguisher during an emergency. In this case, "breaking the glass" means using an account or role with elevated privileges/permissions.

So you "broke the glass" and used this powerful role to restore the database. You're the hero, but what's next?

Post mortem

It is essential to clearly explain the reasons for using this role and the actions performed with it. This is necessary for auditability, as this role can have the capabilities to DELETE data or CREATE additional access. Therefore, it should only be utilized in emergency situations.

It’s important to avoid using this role for everyday tasks. Mistakes can occur; for example, you might think you are working on the staging AWS account and accidentally delete a database, only to realize later that you were actually in the production environment (have you heard how Gitlab did it once?)

In this blog post, you will learn how to establish a break-glass policy and set up Slack alerts for its usage. The remainder of the blog post will delve into technical details.

It's worth noting that implementing Break Glass access is one of the foundational elements of the AWS Well-Architected Framework.

Emergencies or unforeseen circumstances might necessitate temporary access beyond regular permissions for day-to-day work. Having break-glass procedures helps ensure that your organization can respond effectively to crises without compromising long-term security. During emergency scenarios, like the failure of the organization's identity provider, security incidents, or unavailability of key personnel, these measures provide temporary, elevated access beyond regular permissions.

~ AWS Well-Architected Framework: Implement break-glass procedures

How to implement Break Glass in AWS

Case A: IAM Users

If you use IAM users to access your AWS accounts, you can stop using them, configure SSO for your account, and get rid of long-lived credentials.

Case B: SSO

In the context of Single Sign-On (SSO), a Permission Set refers to an Identity and Access Management (IAM) role created on target AWS accounts with specific permissions. This role is intended for users on those AWS accounts.

For example, a single user may have the following roles available for use on the "production" account:

• devops
• devops-break-glass

Each role comes with a different set of permissions. For instance, the "devops" role may have read-only access to databases:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "rds:Describe*",
        "rds:List*",
        "rds:View*"
      ],
      "Resource": "*"
    }
  ]
}

The "devops-break-glass" permission allows users to create, update, or delete RDS databases:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "rds:*",
      "Resource": "*"
    }
  ]
}

Notifications

It's crucial to configure notifications when the break-glass role is used. Those notifications can be used to implement a process requiring a person using that role to explain the need to use it.

There are many ways to achieve that; I would like to show you one based on EventBridge and API Destinations.

1. Create Eventbridge Connection to Slack API

For Authorization key use Slack Bot Token. Eventbridge will store this key in AWS Secrets Manager.

2. Create API destination

For connection ARN, use connection created in step 1.

It will automatically match the authorization method configuration in EventBridge Connection.

3. Create an EventBridge rule to catch all usages of Break Glass roles.

Event Pattern:

{
  "detail": {
    "eventName": ["AssumeRole", "AssumeRoleWithSAML"],
    "eventSource": ["sts.amazonaws.com"],
    "requestParameters": {
      "roleArn": [{
        "wildcard": "arn:aws:iam::*:role/aws-reserved/sso.amazonaws.com/eu-west-1/AWSReservedSSO_break-glass_*"
      }]
    }
  },
  "detail-type": ["AWS API Call via CloudTrail"],
  "source": ["aws.sts"]
}

Target:

• use API destination Type and select the API Destination created in step 2

Input Transformer (very important! Here we are preparing a message for Slack):

{
    "channel": "#break-glass-audit",
    "blocks": [
        {
            "type": "header",
            "text": {
                "type": "plain_text",
                "text": "🚨 Break Glass Role Assumed 🚨",
                "emoji": true
            }
        },
        {
            "type": "section",
            "fields": [
                {
                    "type": "mrkdwn",
                    "text": "*Account:*\n<account>"
                },
                {
                    "type": "mrkdwn",
                    "text": "*Environment:*\nREPLACE_ME"
                },
                {
                    "type": "mrkdwn",
                    "text": "*Region:*\n<region>"
                },
                {
                    "type": "mrkdwn",
                    "text": "*Role:*\n`<role>`"
                },
                {
                    "type": "mrkdwn",
                    "text": "*User:*\n<user>"
                },
                {
                    "type": "mrkdwn",
                    "text": "*Time:*\n<time>"
                }
            ]
        },
        {
            "type": "divider"
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "📣 *This role grants elevated permissions and should only be used in emergencies.*"
            }
        },
        {
            "type": "context",
            "elements": [
                {
                    "type": "mrkdwn",
                    "text": "📝 *Action required:* <user>, please provide a brief justification for using this role."
                }
            ]
        }
    ]
}

And click save.

Now, whenever the "Break Glass" role is assumed, we will receive a Slack notification, it looks like this:

Thanks to using EventBridge, we also get monitoring of how many times a Break Glass role was assumed in the past:

Wait, there's more!

You probably think something is wrong here. A single person can assume an overprivileged role to perform some things, and only after it's done will you see it and ask for an explanation.

According to this principle, it's better to ask forgiveness than for permission.

This solution may or may not fit you, depending on your organisation.

However, having an approval-based process to assume such an overpowered role would be better.

With great power comes great responsibility

So as the next step, we're planning to prepare such an approval-based break-glass:

1. Break glass role assume was requested, notification was sent to Slack
2. Slack notifications include an approval button. If a user or manager with proper permission or who is included in the Well-Architected group clicks "Approve," the user can assume the Break Glass role.

Learn more

• AWS Well-Architected Framework: Establish emergency access process
• AWS Break Glass Roles example implementationare