Terraform: Patterns and Anti-Patterns [Part 1] - Flexible Providers

In the last 15 months, I have become a daily user of Terraform. In that time, several things have happened in my “relationship” to the tool:

• I have used it to build out some complex (as well as some not-so-complex) deployments (and learned a lot of lessons the hard way)
• I have been particularly impressed with the growth in the tool’s capability and HashiCorp’s commitment to pushing the tool forward by expanding its capability set as well as improving core functionality. Just a few examples in this regard:
  • Remote state file locking (at least for S3-stored statefiles) used to require third-party tooling (most folks used terragrunt); now it’s a native capability
  • The ability to pass unserialized-as-string data structures (e.g. lists, maps) to modules used to require a lot of ugly hacks (e.g. serializing everything to a string); now this is first-class functionality
  • The emergence of a robust set of data providers (at least for AWS), e.g. availability zones, IAM roles, EBS volumes, snapshots, etc. Just a year or so ago, getting AZ info dynamically required an independent script masquerading as a module (and even at that, the solution wasn’t truly dynamic).
• I have become a bit of an apologist for Terraform over solutions like CloudFormation (except in very specific use-cases)

While most of my formational time learning and working with Terraform occurred as a consultant with Blue Sentry, I have been very fortunate as of this past May to be working with a much larger user base of Terraform at Capital One. Spanning both experiences, I’ve seen users put their code together in ways that will lead to problems down the road, so I thought I’d put together a set of posts to address these practices, where I’ve seen them lead to problems, and how to architect your code differently to prevent those issues.

The first topic we’ll examine is the provider block. As most of of my experience with Terraform is relative to AWS, that’s the angle I’ll be coming from in these posts. Now, on to the code…

99% of the time when I see some first-time Terraform code, I see a provider block that looks like this:

provider "aws" {
  access_key = "AKIABEXAMPLEKEYID"
  secret_key = "AKIAEXAMPLESECRET"
  region     = "us-east-1"
}

You’re probably wondering what’s so bad about this. I mean, less a few variables, it’s actually a play straight from the Terraform documentation.

Here’s what I recommend instead (let’s ignore variables for the moment):

provider "aws" {
  region = "us-east-1"
}

And here’s why:

• Most people stick this block at the top of main.tf and call it a day, meaning that this file is going straight into a (possibly non-private) repo in Github
• Even if you keep the version with creds in scm in a private repo, you’re still using shared credentials, which opens up a world of other issues (auditing, compliance, etc.)

• If I decide to keep this out of version control for the sake of not wanting to compromise my credentials, other (potentially important) attributes in the provider block aren’t shared across users. These include, but aren’t limited to:

  • max_retries
  • allowed_account_ids
  • forbidden_account_ids

“OK, OK, I get it… ” you say, we need something portable, shareable, and safe to leave in a source code repo. This seems like a happy medium, what about this?

provider "aws" {
   shared_credentials_file = "/path/to/my/.aws/credentials"
   profile = "my-profile"
   region = "us-east-1"
}

If you’re absolutely positive you’re the only person that will ever use this Terraform code and that you have no plans to integrate your code into any sort of CI/CD setup, this will work. Rarely is that the case, though, and the shared_credentials_file/profile setup falls over in this case because:

• Every person/system that could ever potentially use this code must have their credentials file located at the same path and under the same profile key. This is nearly impossible if you have a team of users (where usernames and likely even OSes are different, leading to the path to the shared_credentials_file to never be the same - also, ~ doesn’t work in the provider block) or are using a CI/CD platform (especially one that is owned/managed by someone else).

That brings us back to this:

provider "aws" {
  region = "us-east-1"
}

The use of environment variables (either AWS_PROFILE or AWS_ACCESS_KEY_ID+AWS_SECRET_ACCESS_KEY, with a strong preference towards the former):

• Uncouples our provider implementation from machine, user, and OS-dependent paths and configurations
• Is completely portable across platforms, and is flexible enough to also work in CI/CD contexts (i.e. I’ve never seen a CI/CD system that forbids the use of environment variables)
• Allows us to easily work with profiles that can only be leveraged via STS credentials

A possible danger with this setup is that a user runs this against the wrong account with an AWS_PROFILE set for a different account. Terraform actually provides us with ways to mitigate against this danger, which we’ll cover in our next post.

In the meantime, happy building!