Managing AWS at Scale: Best Practices for Developers

By Mark Studer and Shaun Titus

If you missed Part 1 of our 3-part series for developers and IT professionals where we discussed the many challenges you should address, you can go back to read it here. For our second blog post, we’ll now turn to look at solutions and best practices.

Solutions

Build It Yourself?

At Ephesoft, we are a team of engineers building cloud solutions on top of AWS. The platform provides some great services but tends to lack in connecting those services together into a cohesive and simple experience. AWS focuses on providing the building blocks and they expect the developer to focus on connecting them.

If you’re a small organization with no development skills, then this is probably not an option. However, if you are using AWS, you probably have some sort of technical or development resources available. If those resources are building the platform or service on AWS, ultimately, the quality of that service and the security of it depends on the foundational layers. If you have a poor AWS architecture, those decisions will affect the services you are trying to build and maintain on the platform.

Use Separate Accounts

The first part of our solution is using separate accounts. While we talked about all the challenges you must overcome, using a single account does not allow us to achieve our goals. Single accounts enforce a number of hard restrictions that are not workable for a larger organization. We must have enforced isolation and individual limits. As a service provider, allowing another team or a development environment to break your production environment is not an option. Therefore, having separate accounts is the only way to truly enforce the isolation and separation needed.

What do we need to use separate accounts? First, account structure is critically important, like a naming scheme. Second, we need proper auditing and logging in a centralized manner for distributed accounts, CloudTrail. Next, we need automation and a framework to manage these resources, so we don’t spend weeks or months of IT resources to maintain the system. It needs to handle user provisioning, deprovisioning and role changes. Lastly, as developers, we need to be able to use the CLI and access our different accounts.

Account Structure Best Practices

The first item you need to think about is the account structure and hierarchy. Best practices suggest using a multi-tiered hierarchy and enforcing a matching naming scheme for those accounts. That simple rule will pay huge dividends down the line when you start to look at using other AWS services for management and automating across accounts. We recommend breaking your accounts into 4 tiers: company, business unit or department, project and environment. In scenarios where we just want a generic account, we use placeholders like all Xs to indicate that the specific breakdown is being ignored.

Example Account: AC-MK-ANV-P (Acme Marketing Anvils Production)

Having this naming scheme allows us to easily understand who owns and what is in each account. It also allows us to easily automate account tasks as well as group and rollup accounts at different levels. 

Account Hierarchy: AWS Organizations

Since we’re using multiple accounts, signing up for AWS Organizations is a must. This service allows global AWS admins to structure accounts, consolidate billing, enforce governance and more. There’s no cost and the configuration is straightforward. 

Account Management Made Easy

  • Single “pane of glass” or screen to view all AWS accounts
  • Set account level policies
  • Consolidated billing
  • Account provisioning
  • Additional automation with Control Tower

User Access and Provisioning

Even if the AWS SSO service doesn’t work for your organization, you can still use your existing identity provider for SSO so long as it supports SAML 2.0 along with supporting role-based access groups and cascading permissions sets. Since our IDP supports it, we also provide team and product owners with the permissions necessary to manage the role based groups for their AWS accounts. This helps increase the sense of ownership over accounts and reduces help desk tickets.

Best Practices: Set and Forget 

  • SSO Based (single source of truth)
  • Access via role-based user groups
  • Support cascading permission sets
  • Fully automated role mapping
  • Delegated access control
  • Full auditability of automation changes

User Experience

This solution provides an intuitive account/role selection with a consistent experience between the console and CLI. When a team or product lead adds a user to their group they are automatically provided the proper level access and when the hep desk deprovisions an account, all permissions are automatically revoked.

Simple Account Login: 

  • Intuitive account/role selection
  • Consistent experience between console and CLI
  • Single “pane of glass” or screen to view all AWS accounts and roles available
  • Automatically updates based on IDP group memberships

CLI Access

There are several options for tooling. When we started this journey there wasn’t anything that worked well for us, so we built a tool for using SAML-based SSO in the CLI. It’s an open-sourced PowerShell module on the PowerShell Gallery. It allows you to login your session using SSO to AWS or even manage different profiles to multiple accounts and roles and re-authenticate them on-demand. It uses the Selenium driver to do browser-based authentication allowing it to work with any SAML provider and any set of multi-factor authentication requirements. Our team has been using this tool for a couple years now with great success.  

Since that time AWS has also released a built-in option to the AWS CLI that works if you are using AWS SSO. If you leverage AWS SSO you can get a similar experience from AWS out of the box. In addition, if you are using SAML, there are a couple other community options that allow CLI access when using SSO.

Watch a Quick Demo

Here is a short demo of our tool in action. You’ll notice that some text is blurred out for sensitive information. 

Once you open your terminal, you just enter in Login-AWSSAML and it pops open the browser and directs you to your identity provider. Once you login, it redirects you back to the AWS console to select your account and role that you want. Once you select the role and hit sign in, it pulls your SAML assertion and sends it directly to AWS Security Token Service in exchange for temporary credentials. You can use the CLI as normal in this example. I use the S3 command to list out all the S3 buckets in that account.  

Keep an eye out for the final part of the blog series on the benefits of a multi-account approach.