Bring Infrastructure into Compliance with Composer and Import

Introduction

We’re going to show you a use case for the import feature by applying validations to existing infrastructure. This example walks you through the process step by step. Click here for an overview of the migrate functionality.

Prerequisites

You’ll need to have the Fugue CLI set up and the Conductor installed before you can run this example. If you haven’t done so yet, it just takes a few quick steps.

What we’ll do in this example

Suppose your company has a policy that EC2 instances must receive public DNS hostnames, so all VPCs need to have DNS hostnames enabled.

In order to demonstrate how to bring existing infrastructure into compliance with the corporate policy, we’ll do the following:

  1. Use the AWS CLI to create a noncompliant VPC with an accompanying DHCP options set and subnet.
  2. Transcribe the new infrastructure.
  3. Edit the composition to import the validation that enforces the corporate policy.
  4. Modify the composition to bring it into compliance.
  5. Run the composition with the --import flag to bring the resources under Fugue management.
Bringing Infrastructure into Compliance with Fugue.

Bringing Infrastructure into Compliance with Fugue.

What we’ll have when we’re done

Three resources under Fugue management:

  • A compliant VPC
  • A DHCP options set
  • A subnet

How long it will take

About 20 minutes.

Download

You can download the source code for this example from Github:

If you executed init during the Quick Setup, download the files to the directory where you ran the command.

Get editor plug-ins here.

Let’s go!

Creating noncompliant infrastructure

We’ve provided you with two files:

  • EnableDNSHostnames.lw, a design-time validation that throws an error if a VPC doesn’t have DNS hostnames enabled
  • CreateVPC.sh, a shell script to create a VPC, DHCP options set, and subnet

We’re going to focus on the shell script for this part.

The script uses the AWS CLI to create infrastructure according to the region in the $FUGUE_DEMO_REGION environment variable. Pick any region you like, but regions without a lot of other resources work best.

To set the environment variable, we’re going to use the South America (São Paulo) region:

export FUGUE_DEMO_REGION=sa-east-1

Next, we need to make the shell script executable:

chmod 755 CreateVPC.sh

Now you can execute the script like so:

./CreateVPC.sh

You’ll see output like this:

Creating the following resources with the AWS CLI:
vpc-f08a9394
dopt-f7016093
subnet-73e07f14
All done!

Check out the AWS Management Console in whichever region you’ve selected to visually confirm the new resources.

Here in sa-east-1, we see the VPC...

A new VPC in the AWS Management Console.

A new VPC in the AWS Management Console.

and the subnet...

A new subnet in the AWS Management Console.

A new subnet in the AWS Management Console.

and the DHCP options set.

A new DHCP options set in the AWS Management Console.

A new DHCP options set in the AWS Management Console.

Transcribing the infrastructure

For the next step, we’re going to transcribe the resources we just created. This infrastructure is external to Fugue, so transcribing it is the first step to migrating it.

Composer is part of the Fugue Client Tools, so if you haven’t installed it yet, go ahead and install it now.

Open up Composer and select the Generate Composition button. Select your AWS credentials profile or enter your credentials manually and pick your $FUGUE_DEMO_REGION from the dropdown list.

Composer has an option to filter by AWS service, so under Services to Include, select the following services from the Excluded AWS Services list and move them to Included AWS Services:

  • aws-ec2-dhcpoptions
  • aws-ec2-subnets
  • aws-ec2-vpcs

Because it’s a best practice to tag resources, the shell script added the key “Application” and value “FugueImportDemo” to the infrastructure it created. Composer has an option that allows us to filter by tag key, so we’ll enter Application in the text box labeled “Only include resources that match the following AWS tag.”

Settings for Composer's Generate Composition feature.

Settings for Composer’s Generate Composition feature.

Then, select the Generate Composition button at the bottom, where you’ll be prompted to name the composition and save it wherever you’d like. We’ll name ours TranscribedResources.lw.

Composer generates the composition and then displays it side-by-side with a visualization of the infrastructure:

Composer has generated a composition and a visual representation of the infrastructure.

Composer has generated a composition and a visual representation of the infrastructure.

Tip: If you’d prefer to use Transcriber to generate the composition, you can use the following command:

fugue-transcriber --region $FUGUE_DEMO_REGION -i aws-ec2-vpcs -i aws-ec2-dhcpoptions -i aws-ec2-subnets -k Application TranscribedResources.lw

Whether you’ve used Composer or Transcriber, the composition will look something like this:

composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: sa-east-1
############################################################
region: AWS.Sa-east-1


############################################################
# DHCP Options
#   Generated from AWS resource: dopt-f7016093
############################################################
ec2-dhcpoptions-1: EC2.DhcpOptions.new {
  domainName: "sa-east-1.compute.internal",
  domainNameServers: [
    "AmazonProvidedDNS",
  ],
  region: region,
  resourceId: "dopt-f7016093",
  tags: [
    AWS.tag("Application", "FugueImportDemo"),
  ],
}


############################################################
# Subnet
#   Generated from AWS resource: subnet-73e07f14
############################################################
ec2-subnet-1: EC2.Subnet.new {
  availabilityZone: AWS.A,
  cidrBlock: "10.0.1.0/24",
  defaultForAz: False,
  mapPublicIpOnLaunch: False,
  resourceId: "subnet-73e07f14",
  tags: [
    AWS.tag("Application", "FugueImportDemo"),
  ],
  vpc: ec2-vpc-1,
}


############################################################
# VPC
#   Generated from AWS resource: vpc-f08a9394
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: ec2-dhcpoptions-1,
  enableDnsHostnames: False,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-f08a9394",
  tags: [
    AWS.tag("Application", "FugueImportDemo"),
  ],
}

Note that each resource has the resourceId field. This is the most important Ludwig type for a successful migration. This field contains an identifier that uniquely identifies a resource, and is required in order for Fugue to perform the migration.

If you look in the AWS Management Console, you’ll see that Composer (or Transcriber) has accurately transcribed the three resources we created, right down to the last attribute.

Importing the validation

Now’s the time to find the other file you downloaded, EnableDNSHostnames.lw. It’s a simple design-time validation module that raises an error if the composition importing it has a VPC with the enableDnsHostnames attribute set to False. Move the validation module into the same directory as TranscribedResources.lw, if it’s not there already.

Since this represents the new corporate policy, we’d better make sure our infrastructure is compliant.

Editing required! In Composer, import the validation module to TranscribedResources.lw by adding the following statement after the other import statements (around line 5):

import EnableDNSHostnames

Save your work and Composer will recompile the composition. As expected, you’ll get a validation error because enableDnsHostnames is set to False. Composer marks the erroneous line with a red dot and displays the error message next to the visualization:

Composer displays the composition's validation error.

Composer displays the composition’s validation error.

Here is the full text of the validation error:

ludwig (validation error):
  "/opt/fugue/lib/Fugue/AWS/EC2/Vpc.lw" (line 98, column 12):
  Validations failed:

     98|   let vpc: EC2.Vpc {
     99|             cidrBlock: cidrBlock,
    100|             tags: tags,
    101|             instanceTenancy: instanceTenancy,
    102|             region: region,
    103|             enableDnsSupport: Optional.unpack(True, enableDnsSupport),
    104|             enableDnsHostnames: Optional.unpack(False, enableDnsHostnames),
    105|             dhcpOptions: case dhcpOptions of
    106|                             | None -> Optional(DhcpOptions.defaultForRegionWithTags{region: region, tags: tags})
    107|                             | _    -> dhcpOptions
    108|           }

    - DNS hostnames must be enabled
      (from EnableDNSHostnames.checkDNSHostnames)

  Stack trace:
    In call to new at "TranscribedResources.lw" (line 52, column 12)

(Note: If you are using Transcriber for this example, compile with this command: lwc TranscribedResources.lw)

Editing required! We don’t want our infrastructure to be noncompliant, so we’ll fix the composition by changing the VPC attribute enableDnsHostnames to True (around line 55).

As a bonus, we’re going to change a mutable characteristic in this VPC so it’s easier to see that Fugue is managing it.

Editing required! Change the tag value to "FugueImportDemo-UPDATE" (around line 61).

The new VPC declaration should look something like this:

ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: ec2-dhcpoptions-1,
  enableDnsHostnames: True,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-f08a9394",
  tags: [
    AWS.tag("Application", "FugueImportDemo-UPDATE"),
  ],
}

When you save again, you’ll see that the red dot and error message disappear because the composition has compiled successfully.

Note: We’ve checked the composition for compliance using a design-time validation. If you want Fugue to check the composition’s compliance at runtime, you can load the validation module on the Conductor with the command fugue policy validation-add --name checkDNSHostnames EnableDNSHostnames.lw. Once added, runtime validations are applied to any composition the Conductor runs, not just compositions that import the validation module.

Migrating the infrastructure

It’s time to bring the infrastructure under Fugue management, which means running the composition with the –import flag. Doing a test run first is a best practice, so we’ll add a --dry-run flag:

fugue run TranscribedResources.lw --import --dry-run

This helps us catch any errors that might prevent Fugue from successfully importing the infrastructure.

You should see output like this:

[ fugue run ] Dry running /Users/main-user/projects/TranscribedResources.lw...

Asking Fugue to execute a dry run of composition...

FID:   bf3719c6-33d9-4f2c-9d96-e8bf00535443
JobId: '1511228374'

Requests:
  - aws.ec2.create_tags:
      Region: sa-east-1
      Params:
        Resources:
        - subnet-73e07f14
        Tags:
        - Key:   Fugue ID
          Value: bf3719c6-33d9-4f2c-9d96-e8bf00535443.71e05c26-a39e-5f47-ac21-bf2f694c98a0
        - Key:   Name
          Value: ec2-subnet-1

  - aws.ec2.create_tags:
      Region: sa-east-1
      Params:
        Resources:
        - dopt-f7016093
        Tags:
        - Key:   Fugue ID
          Value: bf3719c6-33d9-4f2c-9d96-e8bf00535443.fb2fc568-d148-5bb8-a75f-206ba53e9e54
        - Key:   Name
          Value: ec2-dhcpoptions-1

  - aws.ec2.modify_vpc_attribute:
      Region: sa-east-1
      Params:
        EnableDnsHostnames:
          Value: true
        VpcId: vpc-f08a9394

  - aws.ec2.create_tags:
      Region: sa-east-1
      Params:
        Resources:
        - vpc-f08a9394
        Tags:
        - Key:   Application
          Value: FugueImportDemo-UPDATE
        - Key:   Fugue ID
          Value: bf3719c6-33d9-4f2c-9d96-e8bf00535443.cbaa1722-2b8b-5aee-b288-b0ef725f065d
        - Key:   Name
          Value: ec2-vpc-1

The output includes a list of AWS API calls that Fugue would make to import each resource. As you can see, there’s a call to aws.ec2.modify_vpc_attribute, which represents the change we made to enable DNS hostnames. There’s also a number of aws.ec2.create_tags calls, which is because when Fugue creates or imports a resource it can tag, it adds a tag with "Fugue ID" as the key and a string representing the Fugue ID as the value.

All looks good, so it’s finally time to import the resources:

fugue run TranscribedResources.lw --import -a importDemo

You should see the usual run output, indicating the CLI has sent the request to the Conductor.

Wait a little bit and then check the status of the process:

fugue status

When the run has successfully completed and has migrated the infrastructure to Fugue, you’ll see status output like this:

Fugue Status Report for user/xxxxxxxxxxxx - Mon Nov 20 2017 8:41pm

State    Updated    Created    Account   FID/Alias      Flags    Last Message
-------  ---------  ---------  --------  -------------  -------  --------------
Running  8:40pm     8:40pm     fugue     importDemo     -e       SUCCEEDED

To visually confirm that Fugue is managing the infrastructure, visit the VPC Management Console and click on the VPC the shell script created.

You should see some changes:

  • The VPC has been named ec2-vpc-1
  • Its DNS hostnames setting says “yes”
  • Its tags include the new Fugue ID
  • Its Application tag has an updated value, “FugueImportDemo-UPDATE”
We migrated this VPC and enabled DNS hostnames.

We imported this VPC and enabled DNS hostnames.

The migrated VPC has newly updated tags.

The imported VPC has newly updated tags.

It’s official – we just brought existing infrastructure into compliance with a corporate policy and enabled it to be managed by Fugue! Since it’s under Fugue’s control now, we can treat the process as we would any other Fugue process. We can get drift notifications, we can update the process, we can apply design-time or runtime validations to it, we can interact with it via the Fugue API Server, we can kill it.

Killing the running process

We’ve finished the demonstration, so we’ll terminate the process and destroy the resources we’ve created:

fugue kill importDemo

When fugue status returns this message, you’re all done:

Fugue Status Report for <user>/<account> - Fri Mar 17 2017 5:31pm

State    Updated    Created    Account    FID/Alias    Flags    Last Message    Next Command
-------  ---------  ---------  ---------  -----------  -------  --------------  --------------
Nothing to see here. Go create something! :-)