Example: Fugue CI/CD with Terraform, GitHub, CircleCI

Note

This is part 1 in a series. For part 2, see Example: Fugue CI/CD with Regula Pre-Deployment Checks.

Our example-tf-circleci GitHub repo shows you how to integrate Fugue into your CI/CD pipeline in order to detect resource misconfiguration and compliance violations as part of every deployment. The example will deploy infrastructure into your AWS account (see remotestate.tf, main.tf), and it incorporates the following components:

  • Amazon Web Services (AWS): Provide cloud infrastructure (a VPC and security group)

  • Terraform: Define infrastructure as code

  • GitHub: Store infrastructure as code in version control

  • CircleCI: Deploy infrastructure via Terraform and kick off Fugue scan

  • Fugue: Scan infrastructure for any noncompliant resources and set a new baseline

_images/cicd-1-header.png

Get Started

You’ll find the full walkthrough in the README. All the code you need is in the repo.

This page includes supplemental information to the walkthrough:

Quick Start

Warning

This example will deploy infrastructure into your AWS account, so make sure you’re OK with the resources you’ll be creating in main.tf and remotestate.tf.

In a hurry? If you don’t need a detailed walkthrough, just follow the steps that require user action:

  1. Create a Fugue environment with the following settings:

    1. Region: us-west-1

    2. Resource types:

      1. AWS.EC2.SecurityGroup

      2. AWS.EC2.Subnet

      3. AWS.EC2.Vpc

    3. Compliance family: CIS AWS Foundations Benchmark (CIS AWS)

  2. Set up GitHub repo

    1. Create and clone a new GitHub repo

    2. Download ZIP of the example-tf-circleci repo and extract files into your own repo

  3. Set up Terraform backend infrastructure in S3

    1. In two places, change the XXXXXXXXXXXX in the bucket name (line 3) to your 12-digit AWS account ID (see note):

      1. {project root}/backend.tf

      2. {project root}/remote_state/backend.tf.backup

    2. From the root directory of your GitHub repo:

      1. cd remote_state

      2. terraform init

      3. terraform plan

      4. terraform applythis will deploy infrastructure into your AWS account (see remotestate.tf)

      5. mv backend.tf.backup backend.tf

      6. terraform init

    3. Move up to parent directory and commit/push changes:

      1. cd .. && git add .circleci/ .gitignore .terraform-version Makefile backend.tf config.mk main.tf remote_state/ scan.sh

      2. git commit -m "Add CI CD example files" && git push

  4. Configure CircleCI environment variables

    1. AWS_ACCESS_KEY_ID (for CircleCI IAM user with AdministratorAccess policy)

    2. AWS_SECRET_ACCESS_KEY (for CircleCI IAM user with AdministratorAccess policy)

    3. FUGUE_CLIENT_ID (for Fugue API)

    4. FUGUE_CLIENT_SECRET (for Fugue API)

    5. FUGUE_ENV_ID (for Fugue environment ID)

  5. Watch the CI/CD magic!

    1. Uncomment line 61 of .circleci/config.ymlthis will deploy a VPC and security group into your AWS account

    2. Commit and push changes:

    3. git add .circleci/config.yml && git commit -m "Uncomment Terraform apply step" && git push

    4. View results in CircleCI dashboard

    5. Approve the hold-for-approval step in the CircleCI workflow to apply the Terraform

    6. Optional: Comment out lines 16-21 of main.tf and commit/push again to fix the compliance violations

    7. Optional: Fix any other compliance violations using the Rule Remediation Steps

List of files in the example

Let’s discuss what each file in the GitHub repo does:

.
├── .circleci
│   └── config.yml
├── .gitignore
├── .terraform-version
├── Makefile
├── README.md
├── backend.tf
├── config.mk
├── main.tf
├── remote_state
│   ├── Makefile
│   ├── README.md
│   ├── backend.tf.backup
│   ├── output.tf
│   ├── remotestate.tf
│   └── variables.tf
└── scan.sh

In the .circleci folder:

  • config.yml: CircleCI configuration file defining the build workflow

In the root folder:

  • .gitignore: List of intentionally untracked/ignored files

  • .terraform-version: Terraform version manager tfenv file – if tfenv is installed, specifies the version of Terraform to use

  • Makefile: GNU Makefile to simplify Terraform commands

  • README.md: Instructions for use

  • config.mk: Configuration information for the Makefile

  • main.tf: Terraform infrastructure as code defining a single VPC and security group; this is what will be deployed in your AWS account via the CI/CD pipeline

  • backend.tf: Not to be confused with remote_state/backend.tf.backup, this backend configuration file specifies where state for main.tf will be stored in S3 (see Step 3: Set up Terraform backend for details)

  • scan.sh: bash script that uses the Fugue API to scan your Fugue environment, report compliance state, and then pass/fail the build accordingly

In the remote_state folder:

  • Makefile: GNU Makefile to simplify Terraform commands

  • backend.tf.backup: The backend configuration file that specifies where state for remotestate.tf will be stored in S3 (similar but not identical to backend.tf in the parent directory; we’ll explain the file extension in a little bit)

  • output.tf: Values to be printed after terraform apply

  • remotestate.tf: Terraform infrastructure as code defining the resources that will make up the S3 backend

  • variables.tf: Variables to specify backend deployment details such as name prefixes

How to create a new CircleCI project

During the walkthrough, you’ll need to add a new CircleCI project. Here’s how to do it:

Log in to your CircleCI dashboard and select Add Projects from the left sidebar. You should see your repo in the list of repositories. Select the Set Up Project button:

_images/cicd-add-project.png

Next, select the Start Building button:

_images/cicd-start-building.png

CircleCI asks whether you want it to add a config.yml file for you, which is unnecessary as our repo already has a configuration file. Be extra careful to select the Add Manually button:

_images/cicd-add-manually.png

You’ll see another prompt double-checking that you already have a config.yml file in your repo. Go ahead and select Start Building:

_images/cicd-start-building-confirm.png

You’ll be taken to the Pipelines page. Select the Project Settings button in the upper right corner.

How to create the CircleCI IAM user

In order to add the AWS access key ID and secret access key to your project settings, you’ll create a CircleCI IAM user in AWS.

Navigate to the Users page of the IAM Management Console and click the Add User button.

Enter a user name – we named ours fugue-circleci-example – and check the box for programmatic access:

_images/cicd-add-user-1.png

For the purposes of this example, we’ve given CircleCI administrative access so it can deploy infrastructure into our account. If you don’t already have an IAM group granting admin access, create a group and assign it the AWS-managed AdministratorAccess policy. (You can also attach the AdministratorAccess policy directly to the user through the Attach Existing Policies Directly button, although this violates CIS AWS 1-16.)

_images/cicd-assign-policy.png

Give the user tags if you like, and then proceed to the review page. If all looks good, click the Create User button.

_images/cicd-add-user-review.png

You should see a “Success” message, along with the new user’s access key ID and secret access key. Copy or download the credentials, then head back to the CircleCI project settings page and select Environment Variables from the left nav. Then select Add Environment Variable.

_images/cicd-add-env-var.png

Follow the rest of the steps in the README.

Line-by-line explanation of configuration

See a line-by-line explanation for the following files:

.circleci/config.yml

We’ll take the CircleCI configuration line by line, but we’ll start at the end, where we define the workflows:

workflows:
   version: 2
   install-and-check-env:
     jobs:
        - terraform-init:
           filters:
             branches:
               only: master
        - terraform-plan:
           requires:
             - terraform-init
        - hold-for-approval:
           type: approval
           requires:
             - terraform-plan
        - terraform-apply-approval:
           requires:
             - hold-for-approval
        - scan:
           requires:
             - terraform-apply-approval

The workflow install-and-check-env described here can be summed up in English like so:

  1. Run terraform init only on commits to the master branch

  2. If the previous job succeeds, run terraform plan

  3. If the previous job succeeds, require manual approval of the Terraform plan before proceeding with the rest of the build

  4. If the plan is approved, run terraform apply without requiring additional human input; this deploys the main.tf infrastructure in your account

  5. If the previous job succeeds, scan the Fugue environment assigned to the environment variable FUGUE_ENV_ID and pass/fail the whole build according to the scan results

That’s the bird’s-eye view of what happens in our CircleCI workflow. Now we’ll take a lower-level look at the individual jobs, starting with the settings at the top of the config file.

# Python CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-python/ for more details
#
version: 2

defaults: &defaults
    docker:
      - image: circleci/python:3.6.1
    working_directory: ~/project
    environment:
      TF_IN_AUTOMATION: true

Above, this sets some basic defaults to be used from job to job. Each job uses a Python 3.6.1 Docker image. The TF_IN_AUTOMATION flag changes the human-readable output to reflect the fact that users don’t directly issue Terraform commands, which is a cosmetic change but nice for logs.

tf-install: &tf-install
  run:
    name: tf-install
    command: |
      wget https://releases.hashicorp.com/terraform/0.12.20/terraform_0.12.20_linux_amd64.zip
      sudo unzip terraform_0.12.20_linux_amd64.zip -d /usr/bin
      rm terraform_0.12.20_linux_amd64.zip
      terraform --version

This command downloads Terraform 0.12.20, unzips it, and moves it to a location within the $PATH, /usr/bin.

For the purposes of this example, we’ve used a stock Docker image that does not have the Terraform binary installed. This means we need to install it during every job in order to use it. To keep the config concise, we’ve defined a YAML anchor &tf-install so we can reuse the tf-install step wherever needed. We do this by referring to its alias *tf-install, which you’ll see further down.

In a real-life scenario, you’d likely want to create your own Docker image with Terraform preinstalled.

jobs:
  terraform-init:
    <<: *defaults
    steps:
      - checkout
      - *tf-install
      - run:
          name: init
          command: terraform init -input=false
      - persist_to_workspace:
          root: ~/project
          paths:
            - .

The first job, terraform-init, installs the Terraform binary (referring to the commands by the alias *tf-install), then initializes the project directory. The files are persisted to the workspace so subsequent jobs can access them.

terraform-plan:
  <<: *defaults
  steps:
    - attach_workspace:
        at: ~/project
    - *tf-install
    - run:
       name: plan
       command: terraform plan -out=plan.tfplan -input=false
    - persist_to_workspace:
        root: ~/project
        paths:
          - .

This command grabs the files persisted from the last job and generates a Terraform plan.

The -out=plan.tfplan flag saves the plan to disk so it can be persisted to the next job.

-input=false means that Terraform won’t request user input for variables if they aren’t already set, since this command is executed in CI/CD.

terraform-apply-approval:
  <<: *defaults
  steps:
    - attach_workspace:
        at: ~/project
    - *tf-install
    - run:
       name: apply
       command: |
         # Uncomment the line below to allow CircleCI to apply the Terraform
         terraform apply -input=false -auto-approve plan.tfplan
         echo Uncomment line 61 of config.yml to let CircleCI apply the Terraform
    - persist_to_workspace:
        root: ~/project
        paths:
          - .

This job is where the Terraform infrastructure as code becomes real infrastructure – with terraform apply.

You can see we’ve commented out the actual command that applies the Terraform using the plan.tfplan file generated in the previous job. This is because you’ll likely want to test out the pipeline before actually creating infrastructure in your AWS account. When you’re comfortable with the setup, you can uncomment the terraform apply line.

-auto-approve skips the usual request for interactive approval, applying the infrastructure without asking for confirmation. The flag is useful for CI/CD scenarios such as this; however, in general it’s important to be very careful when using it, especially in production environments. See Terraform’s recommendations about using it in automation.

scan:
  <<: *defaults
  steps:
    - attach_workspace:
        at: ~/project
    - run:
       name: scan-env
       command: |
         set -e
         bash ./scan.sh

After the Terraform has been successfully applied, this job runs the scan.sh bash script (see details).

scan.sh

We’ll start at the top of scan.sh:

#!/bin/sh

echo "Initiating Fugue scan..."
curl -s -X POST https://api.riskmanager.fugue.co/v0/scans?environment_id=$FUGUE_ENV_ID \
   -u $FUGUE_CLIENT_ID:$FUGUE_CLIENT_SECRET | jq '.' > scaninfo.json

SCAN_ID=$(jq -r '.id' scaninfo.json)

echo "Fugue is now scanning your environment. Scan ID:"
echo $SCAN_ID

These commands use the Fugue API to kick off a scan of the Fugue environment specified in the FUGUE_ENV_ID environment variable. The output of the scan is saved to scaninfo.json, where the scan ID is pulled from and assigned to SCAN_ID. The scan ID is then displayed.

while [ "$(curl -s -X GET https://api.riskmanager.fugue.co/v0/scans/$SCAN_ID -u $FUGUE_CLIENT_ID:$FUGUE_CLIENT_SECRET | jq --raw-output '.status')" == "IN_PROGRESS" ]; do
  printf "Scan in progress...\n"
  sleep 15
done

Every 15 seconds, check the status of the scan. If it’s still in progress, keep checking.

curl -s -X GET https://api.riskmanager.fugue.co/v0/scans/$SCAN_ID -u $FUGUE_CLIENT_ID:$FUGUE_CLIENT_SECRET > scan_results.json

NONCOMPLIANT=$(jq -r '.resource_summary.noncompliant' scan_results.json)

SCAN_STATUS=$(jq -r '.status' scan_results.json)

When the scan is no longer in progress, request the scan results and save to scan_results.json. Assign the number of noncompliant resources to the environment variable NONCOMPLIAINT and the final status of the scan to SCAN_STATUS.

cat scan_results.json | jq '.'

if [ "$SCAN_STATUS" == "SUCCESS" ] && [ "$NONCOMPLIANT" != "0" ]; then
   curl -s -X GET https://api.riskmanager.fugue.co/v0/scans/$SCAN_ID/compliance_by_resource_types -u $FUGUE_CLIENT_ID:$FUGUE_CLIENT_SECRET > scan_results_noncompliant.json
   printf "\nScan completed. Found $NONCOMPLIANT NONCOMPLIANT resource(s):\n\n"
   cat scan_results_noncompliant.json | jq -r '.items[].noncompliant[].resource_id'

   printf "\nUpdating baseline with scan ID $SCAN_ID..."
   curl -s -X PATCH https://api.riskmanager.fugue.co/v0/environments/$FUGUE_ENV_ID -u $FUGUE_CLIENT_ID:$FUGUE_CLIENT_SECRET --data '{ "baseline_id": "'"$SCAN_ID"'" }' > update_baseline.json
   BASELINE_ID=$(jq -r '.baseline_id' update_baseline.json)

   if [ "$BASELINE_ID" == "$SCAN_ID" ]; then
      printf "\nBaseline has been updated."

   else
      printf "\nError updating baseline."

   fi

   printf "\nBuild failed. See scan_results_noncompliant.json for details."
   exit 1

elif [ "$SCAN_STATUS" == "SUCCESS" ] && [ "$NONCOMPLIANT" == "0" ]; then
   printf "\nScan completed. All resources are compliant."
   printf "\nUpdating baseline with scan ID $SCAN_ID..."
   curl -s -X PATCH https://api.riskmanager.fugue.co/v0/environments/$FUGUE_ENV_ID -u $FUGUE_CLIENT_ID:$FUGUE_CLIENT_SECRET --data '{ "baseline_id": "'"$SCAN_ID"'" }' > update_baseline.json
   BASELINE_ID=$(jq -r '.baseline_id' update_baseline.json)

   if [ "$BASELINE_ID" == "$SCAN_ID" ]; then
      printf "\nBaseline has been updated."

   else
      printf "\nError updating baseline. Build failed."
      exit 1

   fi

else
   printf "\nScan error. Build failed. Baseline has not been updated."
   exit 1
fi

This is a long chunk of code, but it’s rather straightforward. We’ll describe it in plain English below.

If the scan succeeded and any noncompliant resources were detected:

  1. Print the number of noncompliant resources along with their resource IDs.

  2. Set the resource configuration recorded in the scan as the new baseline. This facilitates drift detection, should you have notifications enabled.

  3. Fail the workflow so the user knows there were compliance violations in the deployment.

If the scan succeeded and zero noncompliant resources were detected:

  1. Print a success message.

  2. Set the resource configuration recorded in the scan as the new baseline.

  3. Pass the workflow unless the baseline update failed, in which case the overall workflow fails.

If anything else happened:

  1. Print an error message.

  2. Do not update the baseline.

  3. Fail the workflow.

To continue the walkthrough from here, proceed to step 5.

Further reading

Don’t forget to visit the example-tf-circleci repo, which contains the full walkthrough and all of the code in this example.

Next, check out Part 2 of this example, where we add Regula to the front of the pipeline for pre-deployment compliance checks.