Terraform Provider Bundling

Beginning with the 0.10.x version tree of Terraform, HashiCorp decided to decouple providers from the main Terraform runtime (see the 0.10.0 CHANGELOG for details). For a lot of users, this is a seamless win as Terraform will pull down whatever providers it deduces that it needs to execute your build assuming you have network connectivity to reach their servers and grab the files. This flexible architecture allows new providers to be released, bugfixes and features to be introduced in existing providers without requiring a version increment and download of a monolithic Terraform package.

There are, however, scenarios where this actually makes things a little more difficult – for example, behind a corporate proxy that doesn’t have outbound internet access. Such is the case with our CI/CD server context. So, in order to execute our builds in a CI/CD context (Jenkins), we can leverage Terraform’s provider bundling feature to create an artifact that we can store somewhere that our CI/CD server can grab it from (Enterprise GitHub, S3, Artifactory, etc.) that we can unpack in our CI/CD context to give us the runtime environment we need to deploy our infrastructure.

I built my terraform-bundle executable a bit differently from the steps outlined here. Here’s what I did:

  • $ go get github.com/hashicorp/terraform
  • $ go get github.com/hashicorp/go-getter
  • $ go get github.com/hashicorp/terraform/plugin
  • $ go get github.com/hashicorp/terraform/plugin/discovery
  • $ go get github.com/mitchellh/cli
  • $ cd $GOPATH/src/github.com/hashicorp/terraform
  • $ go install ./tools/terraform-bundle

Assuming you now have terraform-bundle in your PATH, let’s manage your build dependencies. At a minimum, I usually use the aws, null, and template providers for a project, so let’s start with a config that captures those providers.

  • First, create a bundle.hcl file in the root of your project. Here’s what it would look like for the providers we just referenced:
terraform {
    version = "0.11.7"

providers {
    aws = ["~> 1.15.0"]
    null = ["~> 1.0.0"]
    template = ["~> 1.0.0"]
  • OK, cool – we now have a text manifest that describes and pins provider versions. Now what? Let’s bundle it!
$ terraform-bundle package -os=linux -arch=amd64 bundle.hcl
  • Running that command produced this output on my machine:
➜  /tmp terraform-bundle package -os=linux -arch=amd64 bundle.hcl
Fetching Terraform 0.11.7 core package...
Fetching 3rd party plugins in directory: ./plugins
- Resolving "aws" provider (~> 1.15.0)...
- Checking for provider plugin on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (1.15.0)...
- Resolving "null" provider (~> 1.0.0)...
- Checking for provider plugin on https://releases.hashicorp.com...
- Downloading plugin for provider "null" (1.0.0)...
- Resolving "template" provider (~> 1.0.0)...
- Checking for provider plugin on https://releases.hashicorp.com...
- Downloading plugin for provider "template" (1.0.0)...
Creating terraform_0.11.7-bundle2018052320_linux_amd64.zip ...
All done!

As you can see, all of the plugins and the core runtime were downloaded and packaged into a .zip file.

To “unbundle” your bundle, copy the .zip into the directory where you’ll be running your terraform plan|apply|... commands, and do something like this:

export platform=linux
export arch=amd64
mkdir -p terraform.d/plugins/${platform}_${arch}
unzip -o terraform_0.11.7-bundle2018052320_linux_amd64.zip -d terraform.d/plugins/${platform}_${arch}
chmod +x terraform.d/plugins/${platform}_${arch}/terraform*
unset platform
unset arch

With this in place, you now have everything you need to run your build!

While a somewhat belt-and-suspenders check, you can also pin your provider versions in your actual Terraform code. Working off the versions we specified earlier for our bundle.hcl file, we’d end up with something like this in one of our “.tf” files (I usually put things like this and my backend config in a main.tf file).

terraform {
    required_version = "= 0.11.7"

provider "aws" {
    version = "~> 1.15"

provider "null" {
    version = "~> 1.0"

provider "template" {
    version = "~> 1.0"

As always, happy building!

comments powered by Disqus