Managing AWS at Scale: Best Practices for Developers

By Mark Studer and Shaun Titus

Amazon Web Services (AWS) makes it very easy for anyone with a corporate credit card to sign up and get started building or implementing software. The challenge is that oftentimes strategic management is overlooked or kicked down the road in favor of fast delivery of a product or service, making it hard to manage and scale AWS. It may not be until AWS adoption grows organically that problems begin to surface. 

In this 3-part series overview, we’ll discuss how to build your application foundation with infrastructure as code (IAC) while addressing common challenges, authentication options, solutions and benefits using AWS. We also examine how to leverage IAC to help you simplify account management and resources as you grow your AWS footprint. 

Topics include:

  • Single vs. Multiple Accounts
  • Isolation and Tracking
  • SSO and AWS SSO
  • Building your own Provisioning System
  • Leveraging the CLI Cost Savings and ROI

Challenges

If you can answer yes to any of the questions here, you’re not alone – many organizations face the same issues.

AWS Account and Identity Challenges

  • Do you use a single AWS account to host both production and test data?
  • Do you have difficulty tracking spending by team and project?
  • Do you suffer from managing lots of Identity and Access Management (IAM) users?
  • Do you have Single Sign On (SSO) with Security Assertion Markup Language (SAML) authentication configured for console access but end up creating IAM users for Command Line Interface (CLI) access?
  • Are you spending hours of time provisioning and deprovisioning AWS access?

Read on to learn more if you answered “yes” to any of the above questions.

Working With What We Have: Single Account and Multiple Accounts

The easiest way to get up and running on AWS is to create a single account and start building. However, a variety of challenges are likely to arise:

  • Having all resources in the same account makes it easy for administrators and developers to make mistakes
  • The sheer number of resources starts to become difficult to manage
  • The rate of change slows out of fear of unknown implications
  • It can be difficult to provide minimum privilege access when all resources are in the same account

Single Account

Let’s look at each of the potential hurdles that you might face with a single account approach: 

1. Resource Isolation

  • Large impact on human-related errors
  • Time spent searching for resources
  • Users lack confidence to make changes
  • Resource-level permissions can be challenging to properly configure and maintain

2. Cost Accounting

It’s common for your finance team to inquire where the spend is coming from. While AWS is always improving on this front, managing cost accounting for a large number of resources in a single account is difficult and in some cases just not possible.  

While you can tag resources for cost reporting, it can be difficult to enforce consistent tagging and not all resources support tags. Cloud costs can become an afterthought for admins and developers who may view cloud spend as someone else’s problem.  

Unfortunately, these challenges make it hard to explain major trends in AWS spend.

3. Resource Limits

Another common problem when using a single account is running up against resource limits. This is especially difficult when multiple teams are sharing a single account and are unaware of the resource footprint for all teams. Bumping up against soft limits can be a waste of time, but reaching a hard limit may require rearchitecting a service or solution or moving it to another account.

Multiple Accounts

The multiple account strategy creates another set of issues, mostly due to complexity. First, you may have many teams using AWS along with multiple projects, multiple environments and a growing number of accounts. Next, you’ll need to consider what resources you want to share between accounts. You might not want to set up the same resources in multiple accounts or need access to common resources. Lastly, how do you get all your uses mapped into these accounts?

That complexity is the reason that the multiple account strategy can add a lot of management overhead to your team. Instead of setting up a single user in your AWS account, your service desk team is trying to juggle onboarding, offboarding and rolling changes across dozens or hundreds of accounts. 

If you have strict isolation rules, there may be a lot of account creation requests. In addition, how do you track the account removals and make sure the resources are actually needed and being properly managed. How do you deal with centralized auditing and logging and meet your compliance requirements? A strategy around how we tackle these issues is needed.

Key Management Overhead For Multiple Accounts:

  • User onboarding and offboarding
  • User role changes
  • AWS account creation/removal
  • Auditing/Logging

Authentication Challenges

Identity and Access Management (IAM) Users

The first option that AWS provides is IAM users. IAM users are the native accounts within AWS and support functionality like groups and roles out of the box. IAM is a great way to get started with AWS for small organizations but it doesn’t scale well with a centralized management process to allow for your IT team to handle provisioning and deprovisioning. Whenever teams choose to use IAM accounts directly they almost always end up with old accounts and users still having access that has been terminated or resigned, which is not ideal. What you end up having is a second user directory when using IAM users which your IT team must manage. 

IAM accounts are simple to use for Command Line Interface (CLI) access assuming you are using a single account. However, when you start using multiple accounts, you either have a bunch of IAM users spread all over the place or a single delegated user that now has problems accessing the CLI and must leverage role switching for the console. 

Role switching is not recommended. The different accounts and roles that you have access to are only stored in your browser as a cookie after you login. And, it only stores the last 5 roles that you have used. Ultimately, you end up with big spreadsheets or documents with links to each of the different accounts and roles.

SSO with SAML

SSO can really help simplify a multiple account strategy by eliminating the need to manage IAM users for console access and the experience is much more user friendly and intuitive than role switching. With SSO configured, users can be assigned roles across AWS accounts from an organization’s identity provider.

However, SSO does bring along some challenges such as CLI access. Managing which users get access to what accounts and roles can quickly overwhelm an IT team. Some identity providers, like G Suite, require each role’s ARNs (which is a long, nondescript, string of characters) to be added to each user’s profile manually. To make matters worse, some identity providers have no built-in mechanism for group based management of these SSO roles.

AWS SSO

AWS has recognized the need for a solution in this space and has added a tool they call AWS SSO. According to the documentation, “AWS SSO configures and maintains all the necessary permissions for your accounts automatically, without requiring any additional setup in the individual accounts.” 

AWS SSO provides a method for SSO with CLI access, removing the need to manage IAM users just for this purpose. This tool is great if you have an identity provider that is supported for auto provisioning with AWS SSO. This is a relatively new service and the AWS team is continuously adding functionality and support to this tool. Unfortunately, the current list of identity providers (IDPs) supported for auto provisioning is still small and in some cases, there are security and cost implications.

Keep an eye out for the second part of our series on how to solve these challenges.